11 Text Encoding for IoT

11.1 In 60 Seconds

Keep One Name Intact Across a Device Update

Picture a sensor label that displays correctly on the bench but becomes damaged text after an update. The team needs to know which boundary changed the bytes.

Firmware means the program stored on a device to control its hardware. A payload means the part of a message that carries the application data.

Send a short name, an accented name, and one invalid byte through old and new versions. Keep source text, encoded bytes, byte limit, firmware version, payload, decoder result, and rejection reason.

This check proves chosen cases, not every language or damaged stream. The deeper sections explain character sets, Unicode, UTF-8, byte limits, validation, and recovery.

Text becomes bytes before it crosses an IoT boundary. ASCII, Unicode, and UTF-8 are not interchangeable labels: ASCII is a small character set, Unicode names characters, and UTF-8 turns those characters into one to four bytes. A safe text field records the allowed characters, the encoded byte limit, the validation point, and what happens when invalid bytes arrive.

11.2 Start With the Story

You will size UTF-8 text fields and validate names, topics, and identifiers before they cross a device boundary. Start by comparing the visible character count with the number of encoded bytes.

Follow the short identifier across four beats to see why character count, byte count, and validation must agree.

11.3 Characters Are Not Bytes

IoT payloads are often described as "text" or "binary," but the real boundary is sharper: text is characters, while transmission and storage are bytes. A device name, a topic path, a JSON key, or a status message must be encoded before it can be sent. The design question is not only "can a person read it?" but also "how many bytes does it take, and how will both sides validate it?"

Three terms answer different questions. ASCII is a small character set for English-oriented text. Unicode assigns a number, called a code point, to every character. UTF-8 encodes those code points as one to four bytes, and it keeps every ASCII character at a single byte.

Think of a label written by two people who never agreed on an alphabet. The same word can occupy a different number of marks for each writer. On the wire, the "marks" are bytes, and only an agreed encoding lets both ends count and read them the same way.

Use Figure 11.1 to follow text from a visible symbol into a packet field. Watch where the representation becomes bytes, because that is what the field limit must count.

Begin at the character panel in Figure 11.1, then follow the code point into its UTF-8 representation and destination field. The lower review boxes separate allowed characters from the byte limit. A name can satisfy the character policy yet still require more storage than its appearance suggests.

In real IoT protocols, that byte boundary appears in ordinary places. An MQTT topic, a CoAP URI path segment, a JSON field name, and a device label shown in a dashboard may all begin as readable text, but the packet, broker, gateway, and database store encoded bytes. A design review should therefore ask two questions separately: which characters are allowed, and how many encoded bytes may the field occupy?

The One-Minute View

Beginner Examples

• An ASCII topic such as lab/device/temp uses one byte per visible character in UTF-8.
• A device label with non-ASCII characters can look short while using more bytes than its visible length suggests.
• A gateway that accepts user-defined identifiers must validate UTF-8 and length limits before forwarding records.

Character Boundary Knowledge Check

Once characters and bytes are separated, text fields can be reviewed like any other protocol field: define the encoding, count encoded bytes, and enforce the boundary before forwarding or storing the value.

11.4 Apply It: Size and Validate Text Fields

UTF-8 does not use one byte per visible character for every string. The encoded length depends on the code points present, so buffers and field limits must be written in encoded bytes.

Make that byte limit explicit in the artifacts that enforce it: firmware constants, gateway validators, broker or API configuration, database schema, and test fixtures. A stable machine identifier can stay deliberately narrow, such as lowercase ASCII plus digits and separators, while a user-facing display name can allow localized UTF-8 under a separate byte ceiling. Keeping those two fields separate prevents a renamed room label from becoming a changed routing key.

The next Apply It: Size and Validate Text Fields decision depends on the diagram Figure 11.2. Reading ASCII against letters, digits, punctuation clarifies the practical meaning of UTF-8 byte width grows with the code point: one byte for ASCII, up to four for supplemental code points.

Read Figure 11.2 from the ASCII row downward, watching the byte boxes extend across each row. Letters, digits, and punctuation in that first row fit in one byte; later groups need more space. Size the encoded field from its actual contents, since a visible symbol does not always occupy one byte.

Worked Example: Counting Bytes

The string IoT is three ASCII characters, so it is also three UTF-8 bytes:

I -> U+0049 -> 0x49
o -> U+006F -> 0x6F
T -> U+0054 -> 0x54

Now mix scripts. A topic with 18 ASCII characters plus 2 Latin-extended characters is not 20 bytes. The ASCII characters are one byte each and the two non-ASCII characters are two bytes each, so the encoded length is 18 + (2 x 2) = 22 bytes. A "20 character" limit would have been wrong.

Where Text Hides in Messages

Topic and Identifier Design

Treat topic names and identifiers as a small protocol, not free-form notes. Prefer a documented character subset for device-controlled identifiers, such as lowercase letters, digits, hyphen, underscore, and slash where hierarchy is needed. Set limits in encoded bytes, and keep a separate display-name field if humans need localized names. Avoid identifiers that change when a room is renamed or a dashboard label is edited, and define how separators, spaces, quotes, and control characters are rejected or escaped.

UTF-8 Byte Sizing Knowledge Check

A field-size decision is complete only when the allowed character set, encoded byte limit, validation point, and stable identifier/display-name split are written into the system contract.

11.5 Under the Hood: Payload Representation and Safe Decoding

The right representation depends on who reads the payload, how constrained the link is, and whether both endpoints share a schema. Text is not the only option, and binary is not automatically smaller in a useful way.

Compare two ways to send a scaled temperature and a status. The binary version is smaller, but only if the receiver already knows the field order, widths, scaling, status-code map, and byte order:

Readable text:
{"temp_tenths":235,"status":"ok"}
Schema-driven bytes:
field 1: signed 16-bit temp_tenths = 235
field 2: unsigned 8-bit status_code = 1

Validate at Trust Boundaries

Use the same checklist for topics, IDs, JSON strings, and other text fields:

1. Identify the allowed character set and the maximum encoded byte length.
2. Encode with the declared encoding, usually UTF-8, then count bytes after encoding.
3. Reject invalid byte sequences on input.
4. Escape or reject control characters and separators that carry protocol meaning.
5. Preserve units, scaling, field order, and schema version when text is converted to compact bytes.
6. Test an ASCII example, a maximum-length example, and at least one allowed non-ASCII example where localization is supported.

Avoid silent repair. Replacing invalid bytes with a placeholder can hide data corruption. For protocol fields, it is usually safer to reject the field and record a validation error than to forward repaired text as if it were correct.

Common Pitfalls

1. Sizing by visible characters. A "20 character" limit is ambiguous unless the allowed characters and the encoded byte limit are both defined.
2. Treating display labels as stable IDs. Human labels change; device identifiers and routing paths should be stable protocol fields.
3. Sending binary without a schema. Small byte arrays are not self-documenting. Record field order, width, signedness, scaling, units, byte order, and version.
4. Accepting invalid UTF-8. Bad byte sequences break parsers and logs. Validate at trust boundaries before storing or forwarding text.

Under-the-Hood Knowledge Check

At this depth, text encoding is part of the data contract. Use explicit encodings, prefer UTF-8 for interoperability, and remember that character count, byte count, and parsing rules are not the same thing.

11.6 Summary

• ASCII, Unicode, and UTF-8 describe different layers of the text-to-byte path.
• UTF-8 keeps ASCII at one byte but uses two to four bytes for many other code points.
• Buffer and field limits must be written in encoded bytes, not visible characters.
• Topic strings and JSON keys can be a meaningful part of small messages.
• Compact and binary payloads need schema discipline, versioning, units, and validation.
• Validate text at trust boundaries, and reject invalid sequences instead of silently repairing them.

11.7 See Also