5 Hash Functions and Data Integrity
5.1 Start With the Fingerprint Question
Change One Byte and Ask Who Protected the Expected Result
Picture a field unit downloading new software and a short check value. A match can show that the bytes equal the expected copy, but it cannot show who chose that copy unless the expected value is protected. The first test changes one byte and names the trust source.
Firmware means the software stored on a device. A gateway means a device or service that joins two system paths. Telemetry means measurements and status sent for remote use.
Hash one known file, change one byte, use an attacker-chosen expected value, repeat an old release, and restart the verifier. Record the input, method, protected reference, version, failure result, and whether install is blocked.
This runway does not prove identity, secrecy, or freshness from a plain hash. The deeper sections separate digests, keyed checks, signatures, key derivation, password storage, migration, and their evidence boundaries.
Imagine a device downloads a firmware image and a short digest that is supposed to prove the image is safe. The digest can tell whether the bytes match an expected value, but it cannot say who chose that value unless the expected value is protected. That is the whole hash lesson in one question: what exactly is being compared, and who made the comparison trustworthy?
This chapter follows the hash family from that question. A plain digest detects change, HMAC adds shared-key proof, a signature protects artifacts many devices must trust, a KDF derives scoped keys, and a password hash slows offline guessing.
5.2 A Fingerprint for Data, Not a Lock and Not a Signature
A cryptographic hash function takes any input, of any size, and produces a fixed-size value called a digest. The same input always yields the same digest, and even a one-bit change to the input produces a completely different digest. That makes a digest a compact fingerprint: a way to tell whether two pieces of data are identical, or whether something changed in transit or in storage. In an IoT system, a digest can catch a corrupted firmware download or prove that a retained telemetry record still matches the bytes the gateway wrote.
The most common confusion is to treat a hash as if it were encryption or a signature. It is neither. A hash has no key and runs in one direction only: you cannot turn a digest back into the original data, so it hides nothing reversibly and provides no confidentiality. And because anyone can compute a hash over data they choose, a bare digest does not prove who produced the data. A hash detects change; it does not, on its own, keep a secret or prove an identity.
If you only need the intuition, this layer is enough: a hash is a one-way fingerprint that detects change but is not encryption and not a signature. To prove a message came from a specific party, add a key with HMAC or a digital signature. To store passwords, use a slow, salted password-hashing function, not a plain digest.
Think of a tamper-evident sticker that shows a unique pattern. If the pattern matches the one you expect, the package was not disturbed; if it differs, something changed. But the sticker alone does not say who applied it, and an attacker who can print their own stickers can reseal a package they opened. That is the gap between detecting change, which a hash does, and proving authenticity, which needs a key or a signature.
Before choosing any hash-based construction, inspect Figure 5.1 to match the input, trust model, and required evidence to the job rather than to digest length.
Read Figure 5.1 from the plain-digest path to HMAC, KDF, and password verification. A digest needs a protected expected value; HMAC adds a shared authentication key; a KDF turns strong secret input and context into scoped keys; and password hashing adds salt and deliberate cost to a weak human secret. The release branch then asks whether the primitive, context, failure behavior, and migration path fit that role. This map supplies the distinctions the rest of the chapter will prove.
5.2.1 The One-Minute View
One-way fingerprint
A digest detects accidental or visible change and lets you compare data, but it cannot be reversed and carries no key, so it is not encryption.
Detecting change is not proving who
Anyone can recompute a plain hash. Proving a message came from a specific party needs a key (HMAC) or a private-key signature.
Pick the construction for the job
Plain digest, HMAC, KDF, and password hashing are different tools. A fast general-purpose hash is the wrong choice for storing passwords.
5.2.2 Beginner Examples
These examples show why different jobs need different kinds of hashes. Comparing a downloaded file against a digest published in a signed release note catches corruption and tampering. The expected digest itself is protected. Appending a plain SHA-256 digest to a message proves nothing about the sender. An attacker who changes the message simply recomputes the digest. A keyed HMAC closes that gap. Storing user passwords as plain fast hashes is unsafe. An attacker who steals the database can guess billions of candidates per second. A salted, deliberately slow password hash is the right tool.
5.2.3 Overview Knowledge Check
If you can explain why a hash detects change but is not encryption or a signature, you can stop here. Continue to Practitioner to choose the right hash-based construction for a real integrity decision.
5.3 Choose the Construction and Protect the Expected Value
The practical job is matching each integrity decision to the right construction and proving the expected value is trustworthy. A plain hash only helps when the digest you compare against already arrives through a protected path; the moment authenticity matters, the design needs a key or a signature. The common review failure is accepting a bare digest as if it authenticated its source. For example, a gateway that accepts an MQTT command from a device should verify a tag over the topic, device identity, sequence or nonce, and payload, not only a SHA-256 value over the payload text.
5.3.1 Walkthrough: From Input to a Trustworthy Check
Follow the sequence from decision to evidence. First, Classify the input and the threat. Decide whether you are checking public data for corruption, authenticating a message between parties that share a secret, deriving keys, storing a password, or verifying a signed artifact. Next, Pick the construction. Plain digest for integrity against a trusted expected value, HMAC for keyed message authentication, a KDF for key derivation, a password-hashing function for verifier storage, and a digital signature for artifacts many devices must trust. Next, Protect the expected value. A plain digest is only as trustworthy as the channel that delivered the digest you compare against, so put it inside a signed manifest or an authenticated protocol. Next, Bind the exact bytes and context. Name precisely which bytes are hashed or covered by the tag, including metadata such as version, identity, and command fields, so nothing security-relevant is left outside. Finally, Test failure closed. Prove that a modified input, a wrong key, a wrong context, a replay, and an unsupported algorithm are all rejected, and that logs record the failure category without leaking secrets.
Inspect Figure 5.2 to distinguish a public mismatch check from a keyed authenticity check before interpreting either value as evidence.
Read Figure 5.2 from the message to the plain digest on the left, then compare the secret-key input and keyed tag on the right. Anyone can replace both a message and its unprotected digest, whereas only a holder of the HMAC key should create a valid tag. The receiver still needs exact byte and context rules and a constant-time comparison. This contrast connects the walkthrough to the chapter’s central rule: choose the construction that proves the required property.
5.3.2 Choosing the Construction
5.3.3 Worked Review: A Firmware Manifest Digest
A firmware package ships an image, a manifest, and a digest, all downloaded from the same server. The team says the digest proves the firmware is genuine. The reviewer turns that into evidence questions.
What the claim covers
The digest reliably detects accidental corruption of the image, and it confirms the image matches the digest that was supplied.
What the claim misses
Because the digest came from the same unauthenticated channel as the image, an attacker can replace both, and the device would happily verify the attacker's image against the attacker's digest.
Conclusion
Put the digest inside a signed manifest or authenticated protocol, verify the signing authority, version policy, rollback counter, and image digest, and add negative tests for a modified image, modified manifest, wrong key, and unsupported algorithm.
5.3.4 Practitioner Knowledge Check
If you can pick the construction and prove the expected value is protected, you can stop here. Continue to Under the Hood for the security properties, the HMAC and KDF mechanisms, and the failure modes.
5.4 Hash Properties, HMAC, KDFs, and Password Hashing
The deeper layer explains the properties a hash must have for security use, why collision resistance is the property that protects signatures, and why HMAC, KDFs, and password hashing are distinct constructions rather than the same hash applied differently.
5.4.1 The Properties That Make a Hash Cryptographic
A general-purpose hash only needs to scatter inputs. A cryptographic hash must also resist three attacks. Preimage resistance means that, given only a digest, it is infeasible to find any input that produces it. Second-preimage resistance means that, given one input, it is infeasible to find a different input with the same digest. Collision resistance means it is infeasible to find any two distinct inputs that share a digest. The avalanche effect ties these together: changing a single input bit flips about half the output bits, so similar inputs do not produce similar digests. Crucially, collision resistance is bounded by roughly half the digest length because of the birthday effect, so an n-bit digest offers only about n/2 bits of collision resistance, which is one reason very short digests are unsuitable for security.
5.4.2 Why Collision Resistance Protects Signatures
Signatures and certificates do not sign a whole document; they sign its digest. If an attacker can find two different documents with the same digest, a signature created over the harmless one is automatically valid for the malicious one. This is exactly why hash algorithms with known practical collisions, such as MD5 and SHA-1, must not be used for new signatures, certificates, or firmware approval, regardless of digest length. Current designs use the SHA-2 family (for example SHA-256) or SHA-3, and a release should name the algorithm its update profile allows and reject anything weaker.
5.4.3 HMAC: Keys Done Right
It is tempting to build a keyed tag as hash(key || message), but with the Merkle-Damgard construction used by SHA-2 that is vulnerable to a length-extension attack: an attacker who sees the tag and the message length can append data and produce a valid tag without knowing the key. HMAC avoids this with a nested, two-pass construction that mixes the key in twice, so it stays secure even with such hash functions. HMAC gives integrity and authenticity to anyone holding the shared key, but because the key is shared it cannot prove which of the two holders produced the tag; that non-repudiation property needs an asymmetric signature.
5.4.4 KDFs Versus Password Hashing
Both transform a secret input into output, but their threat models differ. A key derivation function such as HKDF assumes a high-entropy input, like a shared secret from key agreement, and its job is to extract uniform key material and expand it into separate keys bound to context: protocol, role, identities, and algorithm. A password-hashing function assumes a low-entropy input that a human or factory chose, so its job is to make offline guessing expensive. It adds a unique salt to every entry, which defeats precomputed tables and ensures identical passwords store differently, and it applies a deliberate work factor or memory cost so each guess is slow and resists GPU and custom-hardware acceleration. Functions such as Argon2, scrypt, bcrypt, and PBKDF2 are designed for this; a fast general-purpose hash is not. Using a plain KDF on a weak password does not add this guessing cost, and using a password hash for high-volume message integrity is needlessly slow.
To keep these threat models separate, inspect Figure 5.3 from each input through its transformation and into the evidence retained for review.
Read Figure 5.3 down the KDF path first: high-entropy shared input is bound to roles and transcript context before producing separated keys. Then follow the password path: a low-entropy input receives a unique salt and deliberate work or memory cost before becoming a verifier record. The comparison connects the inputs to different acceptance evidence and shows why swapping the mechanisms does not preserve their security purpose.
5.4.5 Mechanisms and Failure Modes
hash(key||message) tag can be forged.5.4.6 Common Pitfalls
Follow the sequence from decision to evidence. First, Treating a digest as a signature. A plain hash is not authenticity if an attacker can replace both the data and the expected digest. Next, Fast hashes for passwords. Password verifiers need unique salts and deliberate cost; a fast general-purpose hash makes offline guessing cheap. Next, Confusing KDF output with entropy. A KDF derives keys from input material and context; it does not manufacture entropy a weak input lacks. Next, Legacy collision-prone hashes. MD5 and SHA-1 must not back new signatures, certificates, or firmware approval, no matter the digest length. Next, Leaving fields outside the tag. An HMAC or signature must cover every security-relevant field, including routing, identity, and command metadata. Finally, No migration path. Devices need a way to move to stronger algorithms, costs, and salts without disabling validation entirely.
5.4.7 Under-the-Hood Knowledge Check
At this depth, hashing is a family of distinct guarantees built on one one-way function: collision resistance that protects signatures, a protected expected value that makes a comparison meaningful, HMAC that adds a key safely, KDFs that derive scoped keys from strong secrets, and password hashing that makes weak secrets expensive to guess. Name the role, use a current algorithm, and test that bad inputs fail closed, rather than trusting any digest because it looks long enough.
5.5 Pin One Firmware Image to One Digest
For hash integrity, a gateway stores a firmware image whose single changed byte could alter a safety limit. Figure 5.1 follows the image into a fixed-length digest, but Figure 5.2 adds the crucial question of who can create the expected value. Figure 5.3 keeps password stretching separate from both plain hashing and message authentication.
For hash integrity, in this hash-function check, the device may hash the downloaded image and compare it with a signed manifest digest. At the firmware integrity boundary, a matching hash shows the checked bytes match that authenticated expectation. For hash integrity, publishing an unsigned digest beside the image is weaker because an attacker who replaces the image can replace the digest too.
5.5.1 Predict the Integrity Evidence
- Predict: In this hash-function check, two downloads have the same trusted digest. Should the gateway treat their bytes as the same image? Check: Yes, within the security assumptions of the chosen hash and authenticated manifest.
- Predict: A server stores a fast plain hash of a short password. Does digest length stop password guessing? Check: No. A password KDF adds salt and deliberate work to slow each guess.
5.6 Summary
Carry the chapter forward as one connected chain. First, A cryptographic hash maps any input to a fixed-size digest; it is one-way and keyless, so it detects change but is not encryption and not a signature. Then, Security needs preimage, second-preimage, and collision resistance plus the avalanche effect; collision resistance is bounded by roughly half the digest length. Then, A plain digest only authenticates when the expected value is protected by a signed manifest, certificate chain, or authenticated channel. Then, HMAC mixes in a shared secret to give integrity and authenticity, and its nested construction avoids the length-extension weakness of a naive keyed hash. Then, Signatures sign a digest, so a hash with practical collisions (MD5, SHA-1) breaks them; use SHA-2 or SHA-3 for new security decisions. Then, A KDF such as HKDF derives context-bound keys from a strong shared secret; it does not create entropy that a weak input lacks. Finally, Password hashing uses a unique salt and deliberate work or memory cost (Argon2, scrypt, bcrypt, PBKDF2) to resist offline guessing; a fast hash is the wrong tool.
A hash is not encryption. Use a plain digest to detect change against a protected expected value, HMAC to authenticate messages with a shared secret, a digital signature for artifacts many devices trust, a KDF to derive keys from strong secrets, and a salted, costly password hash for stored passwords. Match the construction to the job, and use a current collision-resistant algorithm.
5.7 See Also
See how AEAD pairs confidentiality with a built-in authentication tag, the keyed integrity HMAC provides separately.
Follow how signatures sign a digest, and why collision resistance is what keeps a signature bound to one document.
Connect KDF outputs, HMAC keys, and password verifiers to ownership, derivation context, and rotation.