Math Bridge: Calibration Versus Quantisation

← Back to Sensor Calibration
Math BridgeSensorsStruggle-friendly runway

Is the calibration miss just a digital step?

One thread from a 0.100 °C code step to the chapter's 1.68 °C validation residual.

Phoebe, the physics guidePhoebe guides
The one targetCompare residual with the ADC floor.
The chapter case0.100 °C, scale 1.0817, residual 1.68 °C.
What it buys youRecognise real nonlinearity.

See the relationship before changing it

Set the input, apply the rule, then read the result with its unit.

The retained audit below checks several chapter fixtures. This model keeps those stated values fixed and changes only raw quantization step, so the numeric fixture does not switch without explanation.

Derive the baseline in four named moves

  1. 1

    Name the input. The chapter baseline is 0.1 degrees C.

  2. 2

    Name the relationship. corrected step = raw step x 1.0817

  3. 3

    Substitute with units. 0.1000 x 1.0817 = 0.1082 degrees C

  4. 4

    Read the result. Keep the unit beside the value. Use it only inside the technical boundary on this page.

Predict, then change raw quantization step

Try Predict the direction of corrected step = raw step x 1.0817. Test another raw quantization step, then compare scale-corrected step.

0.1 degrees C
Chapter baseline
Scale-corrected step

Observe Calibration rescales each code step but does not remove residual error. Reset raw quantization step to 0.1 and compare scale-corrected step.

Explain Gain calibration stretches differences between adjacent readings as well as their values, so a larger raw quantization step becomes a proportionally larger corrected temperature step.

Check yourself

What should you do before trusting a moved-control result?
Answer: Predict its direction, apply the shown relationship, keep the units, and reset to the worked baseline.
What does this small model leave out?
Answer: Only raw quantization step moves here. Field effects named in the technical boundary stay fixed.

Honesty note. This teaching model changes only raw quantization step. It is not a field prediction.

1. Separate calibration from resolution

Calibration changes the line used to interpret a reading. Quantisation has already rounded the raw measurement to one of a finite set of steps.

Phoebe: Multiplying a rounded number can move its rungs. It cannot create the missing values between them.

2. Derive the rounding floor

1

One stepThe DHT22 temperature code changes by q=0.100 °C.

2

Error intervalIdeal rounding error lies uniformly between −q/2 and +q/2.

3

RMS floorσq=q/√12=0.0289 °C.

3. Apply the chapter's straight line

The two-point correction is corrected=raw×scale+offset. Its scale of 1.0817 stretches each 0.100 °C raw rung to 0.108 °C, but the ladder is still discrete.

qcorrected=qraw×1.0817=0.108 °C

4. Try a validation residual

σq=q/√12; qcorrected=q×scale; diagnostic ratio=residual/σq

TryMove the validation miss while the chapter's DHT22 resolution, two-point scale, and 0.5 °C HVAC tolerance stay fixed.

Raw code step
Quantisation RMS
Corrected code step
Residual / RMS floor
HVAC tolerance / floor
Outside 0.5 °C tolerance?

ObserveThe chapter's 1.68 °C residual is about 58.2× the unrounded 0.02887 °C RMS floor and exceeds the 0.5 °C application tolerance.

ExplainA discrepancy tens of quantisation floors wide is evidence of model error, nonlinearity, drift, reference error, or conditions—not the last ADC rounding step.

Technical boundaries.

The calibration comparison fixes the scale factor at 1.0817 and assumes evenly distributed rounding with enough signal variation. It leaves out sensor accuracy, repeatability, hysteresis, reference uncertainty, temperature gradients, self-heating, interpolation, rounded coefficients, drift, and correlated noise.

It does not include DHT22 accuracy limits, repeatability, hysteresis, reference uncertainty, temperature gradients, self-heating, interpolation error, coefficient rounding, drift, or correlated noise
Needs separate evidence

Use field evidence or a deeper model before release.

5. Work the chapter's scale

scale=(45.0−0.0)/(43.80−2.20)=1.0817
0.100 °C×1.0817=0.108 °C per corrected rung

6. Diagnose the residual

1.68/0.02887=58.2×
0.500/0.02887=17.3×

The quantisation floor is comfortably below the HVAC tolerance. The straight-line model, not code resolution, is the limiting evidence in this validation case.

7. Check yourself

Does two-point calibration create extra raw resolution?
Answer: No. It remaps existing codes with scale and offset.
Why is 1.68 °C not explained by 0.100 °C steps?
Answer: It is about 58 RMS quantisation floors wide and far beyond the 0.5 °C application tolerance.
What does the large residual prove?
Answer: It proves quantisation alone is not enough; further evidence is needed to distinguish nonlinearity, drift, reference error, or test conditions.
Honesty boundary.

The calibration comparison fixes the scale factor at 1.0817 and assumes evenly distributed rounding with enough signal variation. It leaves out sensor accuracy, repeatability, hysteresis, reference uncertainty, temperature gradients, self-heating, interpolation, rounded coefficients, drift, and correlated noise.

the raw 2.20/43.80 °C points
Raw readings at two calibration reference points
0/45 °C references
True temperatures of calibration reference points
scale 1.0817
Two-point temperature correction multiplier
1.68 °C residual
Remaining error after calibration
0.5 °C tolerance
Allowed temperature error
0.100 °C DHT22 resolution is catalog-typical
Illustrative raw sensor temperature code step

Using the unrounded RMS gives 58.2×; the chapter's 58.1× uses its displayed 0.0289 °C rounding.

Return to the chapter with the maths unpacked →