Carry the Gain in Full, Round Only the Reading

Ada re-derives this chapter’s own numbers step by step, at full precision

ADA · CALCULATION AUDIT

Carry the Gain in Full, Round Only the Reading

A load cell for beehive monitoring reads 410 raw ADC counts at 0 kg and 3685 counts at 50 kg on a 12-bit ADC spanning 0-4095, and the chapter’s own verification check lands the high point at 50.01 kg instead of exactly 50. This audit asks the question that stray 0.01 kg invites: is that gap a property of the load cell, or does it disappear once the gain is carried at full precision instead of the chapter’s rounded 0.01527 kg/count?

Companion to the chapter Lab: Sensor Calibration — every number here comes from that chapter.

Ada: The load-cell example maps 410 counts to 0 kg and 3685 counts to 50 kg, then verifies the high point at 50.01 kg. That 0.01 kg is not the sensor – it is rounding. Let me carry the gain in full and watch it vanish.

• Gain: 50 / (3685 - 410) = 50 / 3275 = 0.01526718 kg/count (the chapter rounds this to 0.01527).
• Offset: 0 - 410 x 0.01526718 = -6.259542 kg (chapter: -6.26).
• Verify the high point with the unrounded coefficients: 3685 x 0.01526718 - 6.259542 = 50.0000 kg exactly, because (3685 x 50 - 410 x 50) / 3275 = 50. The chapter’s 50.01 kg comes only from multiplying 3685 by the truncated 0.01527.
• Resolution equals the gain: 0.01526718 kg = 15.27 g/count; a perfect 12-bit span would give 50 / 4096 = 12.21 g/count.

The 15.27 g is coarser than the 12.21 g ideal by exactly 4096 / 3275 = 1.25x, because the load cell drives only 3275 / 4096 = 80% of the ADC range and wastes the other 20%. Two lessons sit in one example: resolution is set by how much of the converter you actually fill, and coefficients must be carried at full precision through the whole calculation – round the gain early, as the truncated 0.01527 does, and you inject a 0.01 kg error at full scale into every reading it touches.

Every number above is taken from the chapter’s own material and re-derived step by step.