15 Pub/Sub and Topic Routing

Designing Topic Trees, Wildcards, QoS, Retained State, and Routing Evidence

integration-gateways

comm

bridge

pubsub

15.1 Publishers Speak to Topics, Not People

Start with a notice board. A sensor pins a note under a named topic, and anyone with the right subscription sees a copy. The sensor does not know whether the subscriber is a dashboard, an automation rule, or a gateway bridge. That makes pub/sub flexible, but it also means the topic name has to carry the contract clearly enough that the broker can route the right message to the right readers, bound fan-out, review each wildcard filter, and make replay behavior explicit.

In publish-subscribe messaging, a publisher does not name the consumer it is talking to. It publishes a message to a named topic and moves on. A broker compares that topic against the active subscription filters and delivers a copy to each subscriber whose filter matches. The publisher does not know who, if anyone, is listening, and the subscriber does not know who produced the value. That decoupling is the whole point of pub/sub — and it is also where the risk lives.

Contrast this with request-response, where the sender names the receiver and waits for a reply. Pub/sub trades that direct relationship for flexibility: any number of consumers can subscribe to the same data without the publisher changing. The cost of that flexibility is that the topic name becomes the contract. Once dashboards, automations, and bridges depend on a topic, renaming a level or changing its meaning silently breaks them, because nothing on the publisher side knows they exist.

If you only need the intuition, this layer is enough: a broker routes messages automatically, but the engineering work is proving that the right consumers receive the right messages under normal, delayed, replayed, and failed conditions — and that the topic names they depend on are stable.

A simple mental model has five parts: a publisher contract (topic format, payload schema, timestamp, retain and delivery intent), a topic matcher (exact topics and wildcard filters), broker state (active subscriptions, retained messages, queued messages, dead-letter routes), a subscriber contract (filter scope, maximum delivery quality, duplicate handling), and a bridge output that forwards into another protocol, queue, database, or automation.

Pub/sub routing is a broker contract: publishers send once, then the broker matches subscriptions and fans out bounded copies to approved consumers.

Start With the Broker's Job

Topics, not addresses

Publishers send to topics; the broker matches filters and fans out copies. No one names the other side directly.

A topic name is a contract

Once consumers depend on a topic, its level order and meaning cannot change without breaking them.

Matching is not authorization

The real question is not whether a filter matches a topic, but whether the consumer is allowed to receive everything it matches.

Everyday Topic Routes

A meter publishes to plant-a/line-2/meter/mtr-17/power; an energy service subscribes to a filter and receives a copy without the meter knowing it exists.
A new dashboard can subscribe to existing telemetry with zero changes on the device side, which is the convenience pub/sub is built for.
"The broker delivered the message" is not the same as "only the authorized consumers received it and the topic name will not change underneath them."

Overview Knowledge Check

If you can explain why topics, not addresses, connect the two sides, you have the core idea. Continue to Practitioner for topic trees, wildcards, and delivery choices.

15.2 Design Topic Names Like an API

Because topic names are a routing API, design them deliberately. A stable hierarchy puts the most stable routing dimensions first, so that consumers can subscribe to a meaningful slice with a simple filter. A common shape is site / area / device-role / device-id / metric-or-event, giving topics such as plant-a/line-2/meter/mtr-17/power and plant-a/line-2/valve/vlv-04/state.

Five topic-design rules keep the contract healthy:

Routing dimensions first. Put site, area, tenant, or device role early, where consumers filter on them.
Keep volatile values out of early levels. A value that changes often, high in the tree, breaks stable subscriptions.
No secrets, tokens, or personal data in topic names. Topic names are routinely logged, so anything in them can leak.
Version schemas deliberately, in the payload or a chosen topic level, never through accidental naming drift.
Separate telemetry, command, event, and status paths, because they carry different authorization and retention rules.

Wildcard Filters and the Authorization Question

MQTT-style filters use two wildcards: + matches exactly one topic level, and # matches the remaining levels and must be the final segment of the filter. A filter that matches more topics is not automatically better; the decisive review question is whether the subscriber is allowed to receive every topic the filter can match.

Filter

What It Matches

Good Use

Review Question

plant-a/+/meter/+/power

Power readings from any area and any meter in plant-a.

An energy service scoped to one plant.

Is every area in scope for this service?

plant-a/line-2/#

Every topic below one line.

A local gateway bridge for that line.

Is a full subtree really needed, or just a slice?

+/+/gateway/+/status

Gateway status across all sites and areas.

A fleet health monitor with broad rights.

Does a leading wildcard cross tenant or site isolation?

plant-a/line-2/valve/vlv-04/state

Exactly one device-state path.

A targeted automation or troubleshooting view.

Is an exact filter clearer than a wildcard here?

Delivery Quality and Retained State

Delivery quality (QoS) and retained messages are tools, chosen per data flow rather than as a blanket setting. QoS 0 is fire-and-forget, good for high-rate telemetry where the next value soon replaces the last and the consumer tolerates loss. QoS 1 retries until acknowledged, so it can deliver duplicates and the consumer must handle them. QoS 2 adds a handshake to avoid duplication on that exchange, justified only when the extra cost pays for itself. A retained message stores the last value on a topic so a new subscriber immediately receives the current state — ideal for status or a configuration pointer, but a hazard for commands.

Practitioner Knowledge Check

If you can design a stable topic tree, scope wildcard filters to the authorization boundary, and pick delivery quality per flow, you can stop here. Continue to Under the Hood for fan-out, end-to-end delivery semantics, and retained-command risk.

15.3 What the Broker Cannot Promise Alone

The deeper layer is about the things a broker does not solve for you: scaling fan-out, end-to-end exactly-once, and the replay behavior of retained state. Each is a place where a routing design that passed a happy-path test fails under load or after a restart.

Fan-Out and Backpressure

One publication can produce many deliveries, one per matching subscriber. Fan-out is useful, but a broad subscription feeding a slow consumer can build a backlog faster than the consumer drains it. For each topic family, estimate the expected number of matching subscribers, decide what happens when one consumer is slow or disconnected, and put a bounded behavior behind it — a queue limit, a drop policy, or backpressure — rather than letting an unbounded backlog form silently.

QoS Is a Hop Behavior, Not End-to-End Exactly-Once

The most common misconception is that a high QoS guarantees each message is processed exactly once across the whole pipeline. It does not. QoS governs the delivery exchange between the broker and a client; it says nothing about a bridge that retries, a consumer that restarts after partial processing, or a database write that repeats. End-to-end exactly-once effect is an application property, built from a stable message identifier, a source identifier, a timestamp, and a consumer-side idempotency rule that recognizes a repeat and does not act on it twice. Treat QoS as one input to reliability, not the whole answer.

Retained Messages and the Replay Trap

A retained message is delivered to every new subscriber the moment it subscribes. For state — "the valve is open," "the configuration pointer is X" — that is exactly what you want. For a command it is dangerous: a retained command can execute when a device reconnects long after the command was relevant, firing a stale action with no operator intent. Retain state deliberately, and treat any retained command as a special risk that needs an explicit replay and expiry rule.

Dead Letters and Observability

Messages that are malformed, unauthorized, expired, or undeliverable need a defined destination — a dead-letter route — rather than vanishing. And a routing design is only reviewable if it is observable: keep counters for matched messages, dropped messages, retries, dead letters, queue depth, and retained-state changes, so a reviewer can see fan-out and failure behavior instead of guessing.

The Routing Review Record

Record Section

What It Captures

Evidence to Keep

Trap If Missing

Topic taxonomy

Path pattern, stable levels, examples, schema versioning, naming rules.

Sample topics and the naming restrictions.

Topic names drift into a hidden contract.

Filter scope

Allowed and denied filters and the authorization boundary.

A denied-filter test and the access rule.

A broad filter exposes data quietly.

QoS and retain

Delivery quality and retained behavior per data flow.

The choice per flow and its replay consequence.

A retained command replays on reconnect.

Fan-out and failure

Expected fan-out, queue limits, dead-letter and duplicate handling.

Fan-out estimate, replay test, dead-letter counter.

A slow consumer hides a growing backlog.

Under-the-Hood Knowledge Check

At this depth, pub/sub routing is a contract you must keep reviewable as the system grows. Design stable topic names, scope filters to the authorization boundary, bound fan-out, build idempotency for exactly-once effect instead of trusting QoS, treat retained commands as a replay risk, and keep counters so a reviewer can see what the broker is actually routing.

15.4 Build the Route and Watch Who Receives It

15.5 Summary

In pub/sub, publishers send to named topics and the broker delivers copies to matching subscribers; neither side addresses the other, which is the model’s strength and its contract risk.
A topic name is a routing contract: once consumers depend on it, the level order and meaning cannot change without breaking dashboards, automations, and bridges.
Design a stable topic tree with routing dimensions first, volatile values later, no secrets or personal data in names, deliberate schema versioning, and separate telemetry, command, event, and status paths.
Wildcards + (one level) and # (trailing levels, last segment) are about matching; the real review question is whether the subscriber is authorized for everything the filter matches.
Choose delivery quality per data flow: QoS 0 for replaceable telemetry, QoS 1 with duplicate handling, QoS 2 when the handshake is justified; use retained messages for state, not commands.
Fan-out turns one publication into many deliveries, so estimate it per topic family and bound the behavior when a consumer is slow or disconnected.
QoS is a hop behavior, not end-to-end exactly-once; idempotency from a message id, source id, timestamp, and consumer-side rule is what makes repeated delivery safe.
Keep a routing review record and observability counters covering taxonomy, filter scope, QoS and retain rules, fan-out, dead letters, and replay tests.

Key Takeaway

A broker routes messages automatically, but pub/sub design is the discipline of keeping that routing reviewable: stable topic names that act as a real contract, wildcard filters scoped to the authorization boundary, bounded fan-out, and idempotency rather than a QoS level for exactly-once effect. Retain state deliberately and treat retained commands as a replay hazard, then prove the route with publish, subscribe, denied-filter, and replay tests.