IoT Fundamentals · Study deck

Text Encoding for IoT

Picture a sensor label that displays correctly on the bench but becomes damaged text after an update.

Physics Phoebe is your guide for this deck.

datatextencoding

Physics Phoebe, the module guide, in a scene from this chapter.

After studying this chapter

Learning objectives

You will be able to:

Explain: If you only need the intuition, this layer is enough: a character is a symbol, a code point is its Unicode number, and UTF-8 turns that number into bytes.
Explain: The ASCII characters are one byte each and the two non-ASCII characters are two bytes each, so the encoded length is 18 + (2 x 2) = 22 bytes.
Explain: The design question is not only "can a person read it?" but also "how many bytes does it take, and how will both sides validate it?".

Major section

In 60 Seconds

The team needs to know which boundary changed the bytes.

Firmware means the program stored on a device to control its hardware.
A payload means the part of a message that carries the application data.
The deeper sections explain character sets, Unicode, UTF-8, byte limits, validation, and recovery.

Major section

Characters Are Not Bytes

IoT payloads are often described as "text" or "binary," but the real boundary is sharper: text is characters, while transmission and storage are bytes.

A device name, a topic path, a JSON key, or a status message must be encoded before it can be sent.
If you only need the intuition, this layer is enough: a character is a symbol, a code point is its Unicode number, and UTF-8 turns that number into bytes.
ASCII characters stay one byte; many other characters need more.

A character moves through several layers: visible character, Unicode code point, UTF-8 bytes, then a packet or storage field.

Major section

Characters Are Not Bytes (continued)

The design question is not only "can a person read it?" but also "how many bytes does it take, and how will both sides validate it?".

ASCII is a small character set for English-oriented text.
UTF-8 encodes those code points as one to four bytes, and it keeps every ASCII character at a single byte.
The same word can occupy a different number of marks for each writer.

Major section

Characters Are Not Bytes (continued)

Every text field that crosses a device, gateway, or network boundary needs a character policy and a byte limit, not just a character count.

On the wire, the "marks" are bytes, and only an agreed encoding lets both ends count and read them the same way.
The lower review boxes separate allowed characters from the byte limit.
A name can satisfy the character policy yet still require more storage than its appearance suggests.

Major section

Characters Are Not Bytes (continued)

An MQTT topic, a CoAP URI path segment, a JSON field name, and a device label shown in a dashboard may all begin as readable text, but the packet, broker, gateway, and database store encoded bytes.

In UTF-8, ASCII characters keep their one-byte values.
UTF-8A variable-width encoding for Unicode that stores each code point in one, two, three, or four bytes.
Beginner ExamplesAn ASCII topic such as lab/device/temp uses one byte per visible character in UTF-8.
A device label with non-ASCII characters can look short while using more bytes than its visible length suggests.

Major section

Apply It: Size and Validate Text Fields

UTF-8 does not use one byte per visible character for every string.

The encoded length depends on the code points present, so buffers and field limits must be written in encoded bytes.
Keeping those two fields separate prevents a renamed room label from becoming a changed routing key.
Letters, digits, and punctuation in that first row fit in one byte; later groups need more space.

UTF-8 byte width grows with the code point: one byte for ASCII, up to four for supplemental code points.

Major section

Apply It: Size and Validate Text Fields (continued)

Size the encoded field from its actual contents, since a visible symbol does not always occupy one byte.

A topic with 18 ASCII characters plus 2 Latin-extended characters is not 20 bytes.
A "20 character" limit would have been wrong.
Structured payloadsJSON keys and string values are UTF-8 text.

Major section

Apply It: Size and Validate Text Fields (continued)

The ASCII characters are one byte each and the two non-ASCII characters are two bytes each, so the encoded length is 18 + (2 x 2) = 22 bytes.

Where Text Hides in MessagesIdentifiers Device IDs, asset names, room names, and user labels need a documented character policy and a byte limit.
Topics and pathsTopic strings and URI paths can repeat on every message, so descriptive names should be chosen deliberately.
A field name can be larger than the sensor value it labels.
DiagnosticsText logs help review, but production messages still need size limits and escaping rules.

Major section

Under the Hood: Payload Representation and Safe Decoding

The right representation depends on who reads the payload, how constrained the link is, and whether both endpoints share a schema.

Constrained systems that still need flexible field structure.
Smallest when fields and units are fixed.
Replacing invalid bytes with a placeholder can hide data corruption.

IoT payload representation comparison contrasts readable text, compact structured formats, and schema-driven binary formats by inspection ease, byte overhead, parser and schema needs, and best-fit situations. It recommends choosing against payload budget and operational inspection requirements rather than a named format ranking.

Major section

Under the Hood: Payload Representation and Safe Decoding (continued)

For protocol fields, it is usually safer to reject the field and record a validation error than to forward repaired text as if it were correct.

Common PitfallsSizing by visible characters: A "20 character" limit is ambiguous unless the allowed characters and the encoded byte limit are both defined.
Treating display labels as stable IDs: Human labels change; device identifiers and routing paths should be stable protocol fields.
Sending binary without a schema: Small byte arrays are not self-documenting.
At this depth, text encoding is part of the data contract.

Major section

Summary

UTF-8 keeps ASCII at one byte but uses two to four bytes for many other code points.

Buffer and field limits must be written in encoded bytes, not visible characters.
Topic strings and JSON keys can be a meaningful part of small messages.
Validate text at trust boundaries, and reject invalid sequences instead of silently repairing them.

Deck summary

Key takeaways

The team needs to know which boundary changed the bytes.

IoT payloads are often described as "text" or "binary," but the real boundary is sharper: text is characters, while transmission and storage are bytes.
The design question is not only "can a person read it?" but also "how many bytes does it take, and how will both sides validate it?".
Every text field that crosses a device, gateway, or network boundary needs a character policy and a byte limit, not just a character count.
UTF-8 does not use one byte per visible character for every string.

Retrieval practice

Recall check 1 of 4

Physics Phoebe says: answer from memory, then check your reasoning.

Q1A device sends plain ASCII labels today but may later include non-English characters. What should the designer decide early?

AAssume every visible character is exactly one byte forever

BStrip all non-English text from the system design

CChoose an encoding only after packets fail in the field

DChoose a defined encoding such as UTF-8 and account for byte length.

Show answer

Answer: D Text encoding is a payload design decision because characters and bytes are not always the same thing.

Retrieval practice

Recall check 2 of 4

Physics Phoebe says: answer from memory, then check your reasoning.

Q2A topic has 18 ASCII characters and 2 allowed Latin-extended characters, where each Latin-extended character uses 2 UTF-8 bytes. What is the minimum encoded byte length?

A20 bytes

B22 bytes

C36 bytes

D80 bytes

Show answer

Answer: B 18 ASCII bytes plus 2 characters at 2 bytes each is 18 + 4 = 22 bytes.

Retrieval practice

Recall check 3 of 4

Physics Phoebe says: answer from memory, then check your reasoning.

Q3A device puts user-entered text into a routing topic. The text contains a separator that changes topic matching. What should the gateway do?

AForward the text unchanged because it displays correctly.

BEscape or reject the separator under the topic character rule.

CRemove every separator without recording the change.

DTreat the display label as a permanent device identifier.

Show answer

Answer: B A protocol separator must not silently change the route chosen for the device.

Retrieval practice

Recall check 4 of 4

Physics Phoebe says: answer from memory, then check your reasoning.

Q4A gateway receives a device identifier that contains an invalid UTF-8 byte sequence. What is the safest handling for a protocol field?

AReplace the invalid bytes with a placeholder character and forward it

BReject the field and record a validation error rather than forwarding repaired text

CForward the raw bytes unchanged and let the cloud sort it out

DTruncate the identifier to its first valid byte and continue

Show answer

Answer: B Silent repair can hide corruption; rejecting and logging keeps a bad identifier from propagating as if it were valid.

Print reference

Answers

Answer key.

D · Text encoding is a payload design decision because characters and bytes are not always the same thing.
B · 18 ASCII bytes plus 2 characters at 2 bytes each is 18 + 4 = 22 bytes.
B · A protocol separator must not silently change the route chosen for the device.
B · Silent repair can hide corruption; rejecting and logging keeps a bad identifier from propagating as if it were valid.