MQTT Python: Reconnect and Backoff Timing: Reference
More
Depth: Practitioner
Progress saves as you read. Mark sections complete as you go; your progress appears in the sidebar.
DRAFT — practice exemplar.
A lost MQTT connection can delay a sensor reading even after a client tries to reconnect. This chapter follows the waits between retries and the path of a message to its receiver. You will calculate delays, trace callbacks, and look for proof that a reading arrived.
Start With the Decision
Imagine a room sensor losing its broker connection while a dashboard waits for a reading. The client reconnects and prints a success message. Has the reading reached the dashboard? You will need two kinds of evidence: a retry trace and a receiver’s output.
A broker routes published messages to matching subscriptions. Backoff makes the wait between failed attempts grow. A cap limits each wait. Jitter varies waits so devices need not retry together. First, predict a capped delay in the check. Keep your answer before opening its feedback.
Route Overview
This is part 2 of 2. MQTT Python: Reliable Client Patterns supplies the callback and network-loop pattern. Use it if you need to separate connecting, subscribing, and receiving before attempting the exercises.
Learning Objectives
- Calculate exponential retry delays with caps and jitter.
- Trace MQTT callbacks and reconnect state on an ESP32, and identify the separate TLS setup.
Chapter Roadmap
Work from a delay prediction to a missing-message diagnosis. Write each prediction before using the tool or opening the worked feedback. Later tasks give fewer hints.
- Try It: Exponential Backoff Timing Visualizer — easy: predict waits, then change the cap.
- Checkpoint: Reconnect and TLS Timing — separate waiting from connection work.
- Try It: MQTT Callback Patterns on ESP32 — intermediate: trace the retained sketch.
- Try It: ESP32 MQTT with Reconnection and Callbacks — compare a connection print with a message callback.
- MQTT Debugging Flow — choose the next observation for an empty dashboard.
- Try It: MQTT Debugging Checklist Simulator — test a hypothesis against a suggested diagnosis.
- Interactive Calculators — transfer: identify assumptions that need measurements.
- Checkpoint: Production Trade-Offs — explain what a model output can support.
- Common Pitfalls — connect each mistake to a missing or misleading observation.
- 1. Running MQTT Without TLS — keep protection while diagnosing timing.
- 2. Ignoring Last Will and Testament Configuration — distinguish broker detection from device health.
- 3. Using a Single MQTT Connection for High-Throughput Publishing — measure before partitioning.
- Label the Diagram — retrieve the client, callback, and loop responsibilities.
- Code Challenge — distinguish local handler registration from broker subscription.
- Order the Steps — explain what each added behavior contributes.
- Match the Concepts — pair an event with its handler.
- Deep-Dive Note: Python Client Loop Boundaries — trace work across a reconnect.
- Lab: First MQTT Message — prove a synthetic reading reached your subscriber.
- Summary — compare your retry trace with your receipt record.
- Knowledge Check — diagnose before reading the explanation.
- Quiz: MQTT Python Patterns — transfer to callback registration and TLS timing.
- Concept Relationships — find the mechanism behind a failed prediction.
- See Also — choose a follow-up by the evidence you still need.
- What’s Next — apply the same tests to another client or broker.
- Key Takeaway — state the retry policy and the proof of receipt.
Checkpoint: Reconnect and TLS Timing
Broker Bex asks: “What did your timer actually count?” A delay sum describes waiting. A connection trace must also account for attempts, TLS setup where used, and subscription state. A receiver log tells you whether a reading arrived. Keep those observations separate when choosing a retry policy.
Try It: MQTT Callback Patterns on ESP32
Intermediate practice: trace before running. The sketch has a function named callback and a function named reconnect. Predict which one prints “Connected!” Where does the retry counter start, and what happens after its limit is reached?
MQTT Debugging Flow
Intermediate practice: choose an observation. A publisher gets a broker acknowledgement, but the dashboard stays empty. Write what that acknowledgement establishes. Then name the next observation that could separate a topic mismatch from a subscriber that is not processing packets. Follow the same publication across both delivery legs.
For a boiler publication, PUBACK confirms the publisher-to-broker exchange. The acknowledgement does not show that the receiving application received or decoded the value, or acted on it. Check the subscriber output and its topic filter next. A second matching subscriber can help you compare what each receiver sees.
Follow the route in order, recording what each tool actually sees:
- Check connection: Use the device’s Serial Monitor to inspect Wi-Fi and MQTT status. Keep a connection print separate from a publish result.
- Verify message flow: Connect MQTT Explorer to the same broker with an authorized matching subscription. Its view shows traffic delivered to that connection.
- Validate topics: Use
mosquitto_subon the intended topic branch. Compare the actual published topic with the dashboard’s actual filter. - Test payloads: If the payload contract is JSON, validate that JSON and its encoding. A received publication can still fail application decoding.
Worked feedback: an acknowledgement cannot repair a filter
The acknowledgement confirms the publisher-to-broker exchange. Compare the dashboard’s stored filter with the concrete published topic. Then check whether its network loop runs and its handler is registered. A mismatched filter gives the broker no matching route to that dashboard. A stopped loop leaves incoming work unprocessed. Longer retry waits do not correct either condition.
For the production branch, use the Mosquitto 2.0.22 configuration reference to review the listener, TLS, password-file and ACL settings together. The selector supplies a review checklist, not a deployment procedure. The password utility manual documents that -c is for a new file and overwrites an existing file.
Interactive Calculators
Transfer practice: decide what to measure. The missing-reading problem now involves a fleet. For each calculator, identify a configured input, a model assumption, and an observation you would need before using the result. Write those three items before moving a control.
These are scenario worksheets. A calculated number is evidence about the formula, not a measured broker limit, board benchmark, battery test, or incident forecast.
Broker Connection Capacity Planner
Before using the planner, ask whether each device really needs the assumed connection allowance. The code adds 20 percent to device count, assumes two connections per backend service and three per monitoring tool, then applies the separate Headroom control to that subtotal. The memory, descriptor, and alert values are assumptions too.
Predict the default subtotal from 500 devices, 5 backend services, and 2 monitoring tools. Then check it with the tool.
Worked feedback: two allowances appear in this model
The device allowance is 600 connections. Backend services add 10 and monitoring adds 6, giving 616 connection slots. Applying the separate 20 percent headroom and rounding up gives 740 slots. That is the worksheet’s output. It is not proof a broker can sustain that many connections or their reconnect burst.
TLS Handshake Timeout Estimator
Choose what evidence would justify extending a timeout while keeping TLS enabled. This worksheet uses illustrative device constants, adds a network component and an assumed broker delay, then applies a chosen margin. Its resumption estimate assumes half the modeled handshake time. These values are not board benchmarks.
Compare its output with what a real test would need: handshake duration on the target board, certificate checks, the actual link, and broker load. Check the chosen library’s timeout API and units before changing firmware.
Public vs Private Broker Risk Calculator
Which inputs need evidence before you could use this model to choose a broker? The code uses made-up costs and monthly chances of harm. These are not incident data or prices from a provider. The formula assumes that risk stays the same each month. It also assumes that one month’s risk is independent of another’s. The difference subtracts the assumed private-broker cost from the loss in this scenario. It does not measure how much risk a private broker removes.
Changing the duration tests the math. It cannot show that the risk inputs are valid or set a security budget.
Battery Life: loop_forever() vs loop_start()
Would changing a Python loop API establish a battery saving? Both loop_forever() and loop_start() wait for socket activity. One runs in the calling thread; the other uses a background thread. That difference alone supplies no current measurement.
The worksheet compares assumed steady and duty-cycled currents. Dividing milliamp hours by milliamps gives idealized hours. Use measured board current for an engineering comparison. Its alternate-current and active-time assumptions do not follow from choosing a Paho method.
MQTT Message Delivery Reliability Calculator
Before changing QoS, predict whether the subscriber leg can have a stronger contract than the publication it receives. Then use the worksheet to examine an illustrative outage model. The selected loss percentage is an assumed fraction of missed publications, not a measured TCP packet-loss rate.
The model also assumes the broker grants the requested subscription QoS. A real test must record the granted value.
Worked feedback: what the QoS calculation assumes
The subscriber leg is bounded by the publication QoS and the broker-granted subscription QoS. This model uses the requested subscription value as though it were granted. It assumes all eligible QoS 1 or 2 exchanges finish after reconnection. It also uses the selected outage percentage as a hypothetical QoS 1 duplicate rate.
Those assumptions explain the displayed count. Actual loss and repeats depend on session state, buffering, broker limits, and retries. To explain the missing reading, inspect those states and the receiver log. A TCP packet-loss percentage does not supply a publication-loss percentage.
Review the Production Evidence
Compare each model result with a live observation. A predicted delay or capacity is useful only when its assumptions fit the test.
Checkpoint: Production Trade-Offs
Broker Bex asks: “Which value did we observe, and which did the worksheet assume?” Capacity needs a load test. TLS timing and current need target-device measurements. A subscription needs its granted QoS checked. For the missing reading, keep the model prediction beside the actual received topic and payload.
Explain One Failed Prediction
Explain one wrong prediction before trying the retrieval tasks. Was the missing fact a wait, a connection state, a matching subscription, or a running handler?
Common Pitfalls
For each mistake, name the observation that would expose it. Then compare your correction with the panel. A client that reconnects can still be unprotected, report stale health, or queue work faster than it can send it.
A plaintext connection exposes credentials and payloads to observers on the network path. A successful demo therefore does not prove the traffic is protected. Use TLS 1.2 or later with identity checks, authentication, and topic authorization for production (RFC 8996). If TLS setup times out, measure that setup and adjust supported timeouts with margin. Removing TLS changes the protection rather than diagnosing the delay.
A retained “online” message can outlive an unexpectedly lost client. Configure a Last Will when the application needs the broker to report that loss. Test when the broker detects it and publishes the will. That status reflects broker detection; it is not immediate proof of the device’s physical health. After reconnecting, test the subscription and a fresh received reading too.
A connection writes through one TCP socket. Acknowledgements, in-flight limits, and backpressure can create queues, depending on payloads, the link, and broker capacity. Measure sustained rate and publish latency before adding connections. More connections also change the capacity and reconnect load you must test.
Label the Diagram
Place the client, background network thread, connection callback, message callback, and retry timer. Trace each broker event to the code that handles it.

Knowledge check · self-test
Knowledge check
Test your understanding.
Code Challenge
Complete both handler registrations before opening the explanation. Then answer a separate question: does registering a local function also ask the broker to send that topic?

Knowledge check · self-test
Knowledge check
Test your understanding.
Worked feedback: two routing decisions
message_callback_add() selects a local handler for a matching received topic. subscribe() requests broker traffic. The temperature and humidity handlers belong on their respective filters; the other method names in the choices are not the API demonstrated here. Without a matching subscription, local handler registration alone does not create a broker route.
For reconnection, move subscription setup into the connection callback as the lab subscriber does. The short challenge tests registration; it is not a complete reconnect implementation.
Order the Steps
Arrange the retained progression. Explain what failure each addition helps you detect or handle. Reaching the last card still requires tests of the actual application.

Knowledge check · self-test
Knowledge check
Test your understanding.
Match the Concepts
Match each method or callback to its job. A callback is a function run for an event. In the missing-reading case, separate a lost link from a received message. Then identify the work that sends packets.
Deep-Dive Note: Python Client Loop Boundaries
Transfer practice: trace the next connection. A client already has its message handler. After reconnecting, it still needs the appropriate subscription state and a running network loop. Write where each responsibility belongs before checking the lab subscriber.
Assign handlers before connecting and starting the loop so early packets have a handler available. loop_forever() keeps the calling thread in network service. loop_start() runs that work in a background thread. A manual loop needs regular service from the application.
A slow database write or other long task inside on_message delays other network-loop work. Validate the message, put slow work on a queue, and let worker code handle it. A callback assignment cannot compensate for a loop that is starved.
MQTT 5 adds state with different lifetimes. Topic aliases map topic strings to integers for one connection; rebuild that mapping after reconnecting. PUBLISH user properties travel with a message, including retained delivery. The fragment sets a schema property. It assumes you already have an MQTT 5 client and a payload.
from paho.mqtt.properties import Properties
from paho.mqtt.packettypes import PacketTypes
props = Properties(PacketTypes.PUBLISH)
props.UserProperty = ("schema", "v3")
client.publish(
"site/plant-a/line-3/telemetry/temperature",
payload,
qos=1,
properties=props,
)
For the reconnect trial, record callback assignment, subscription setup, loop service under load, and the first received message. If using MQTT 5, check per-connection aliases separately from message properties. Keep slow application work out of the network callback.
Lab: First MQTT Message
Use this lab to trace a reading from a publisher to a subscriber. Compare both outputs to see what evidence a live test would need.
Trace the room-reading story through the Python code below. Both programs name the same broker and topic. The numbered steps also describe a proposed live exercise; they have not been validated with installed Paho or a real broker. For a live trial, matching payloads in both outputs would be evidence to collect, not a promised result.
Duration: 45 minutes allocated; completion time is not measured. Difficulty: beginner setup, followed by intermediate diagnosis and transfer.
Learning objectives:
- Connect a Python client to the practice broker.
- Subscribe to the exact topic used by the publisher.
- Match a received payload with the publisher’s actual output.
- Predict a wrong-topic or offline result, then compare it with observation.
Materials and prerequisites:
- A computer with Python 3.8 or later and basic Python knowledge.
- Paho MQTT 2.0 or later for
CallbackAPIVersion.VERSION2. - Two terminals, a text editor, and an internet connection for the live exercise.
Version tested: Python 3.13.2 with an in-memory Paho stub, offline only. Date: 2026-10-25. The subscriber, publisher, and displayed backoff fragment executed with socket/DNS access blocked. Tests covered retry waits, success/exhaustion, a transport exception, wrong-topic correction, matching synthetic payloads, and publisher cleanup after success or failure. Paho is not installed in that environment. These checks exercise Python control flow, not Paho networking or a real broker. No live broker availability, connection, subscription, receipt, or recovery result is claimed. Record your actual Python, Paho, and broker versions and test date with the live results.
Use synthetic values only. The retained code connects to the shared plaintext test service on port 1883. A distinctive topic separates exercises but does not make data private. Availability and unrelated traffic can affect a public test run.
First trace the numbered steps offline. Output descriptions below identify print sites and conditions in the code; they are not a live transcript or a guarantee of delivery. If attempting the live exercise, save broker address, topic, payload, send and receipt times, and both outputs. A matching receiver record, a diagnosed failure, and a successful retry after correction would be required to validate that trial. Stop only programs you started.
Step 1: Environment Setup (10 min)
Predict what the import check can establish. Run it in the same Python environment you will use for both programs. If the import fails, check that pip installed into that environment. The success string proves an import, not a broker connection or the availability of the required callback API version.
# Install the MQTT client library
pip install paho-mqtt
# Verify installation
python -c "import paho.mqtt.client as mqtt; print('MQTT library ready!')"
Step 2: Connect to Public Broker (10 min)
Create mqtt_subscriber.py with the retained code. Before running it, find the callback assignments, the subscription, and the call that keeps networking alive. Predict which print happens during setup and which requires an incoming message.
import paho.mqtt.client as mqtt
# Callback when connected
def on_connect(client, userdata, flags, reason_code, properties):
print(f"Connected with result code {reason_code}")
# Subscribe to a test topic
client.subscribe("iotclass/lab1/temperature")
# Callback when message received
def on_message(client, userdata, msg):
print(f"Received: {msg.topic} -> {msg.payload.decode()}")
# Create client and set callbacks
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
client.on_connect = on_connect
client.on_message = on_message
# Connect to public broker (test.mosquitto.org)
client.connect("test.mosquitto.org", 1883, 60)
# Start listening
print("Waiting for messages... Press Ctrl+C to stop")
client.loop_forever()
The code uses iotclass/lab1/temperature. Other learners may use it too, so a message can arrive before your publisher runs. Replace lab1 with a distinctive test suffix, such as ana-7342, in both programs if you need to separate your trial. Record the actual resulting topic. Topic spelling is not an access-control rule; use only synthetic values.
Predict the callback signature and the method that requests a subscription, then fill both blanks. Compare the signature with the subscriber you just created.

Knowledge check · self-test
Knowledge check
Test your understanding.
Packet-size practice: This is a small MQTT 3.1.1 QoS 0 example. It uses a two-byte fixed header because Remaining Length is below 128 bytes. It is separate from the lab publisher, which already uses QoS 1.
Predict the packet size for the 25-byte ASCII topic iotclass/lab1/temperature and the 5-byte ASCII payload 25.3C. Include the two-byte topic-length field. Try the calculation before opening the answer.
Try it yourself:
Worked feedback: count the fields, then the fleet
The fixed header adds 2 bytes. The topic-length field adds 2 more. Adding the 25-byte topic and 5-byte payload gives 34 bytes per MQTT packet. Counting topic and payload alone gives 30 and misses both header fields.
At a 15-minute interval, each sensor publishes 96 times a day. For 500 sensors, 500 times 96 times 34 gives 1,632,000 bytes a day, or 1.632 decimal megabytes a day. With the example’s assumed price of ten cents per megabyte, that is 0.1632 dollars a day. Over 30 days it is 4.896 dollars, rounded to 4.90 dollars.
This counts the modeled MQTT packets, not TCP, IP, TLS, or link overhead. It is not a cellular bill. The retained calculator uses string length and a fixed header size, so keep its inputs to short ASCII strings for this exercise. Arbitrary Unicode text or larger packets need a byte-aware calculation.
For the changed case in the check, a 19-byte topic and 4-byte payload total 27 bytes after adding the same two header fields.
Run the subscriber: python mqtt_subscriber.py.
If execution reaches the setup print, it prints “Waiting for messages… Press Ctrl+C to stop”. If Paho invokes the connection callback, its body prints “Connected with result code” followed by the supplied result, then calls subscribe(). Inspect that result rather than requiring the literal character 0 in its formatted text. Calling subscribe() alone does not establish a granted subscription.
If no connection result appears, check broker address, reachability, and whether the loop is running. If the broker refuses the connection, inspect its reported reason. A successful connection without a received message still leaves topic matching and sender activity to test.
Step 3: Publish Messages (15 min)
Create mqtt_publisher.py. Before running it, predict how many readings its loop generates and which part waits for the publish exchange. Keep the subscriber running first.
This publisher already uses QoS 1. The subscriber’s plain subscribe() call requests QoS 0 (Paho API). The publisher’s acknowledged exchange therefore does not establish acknowledged delivery on the subscriber leg (MQTT 3.1.1 §§3.8.4 and 4.3.2).
import paho.mqtt.client as mqtt
import time
import random
client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2)
client.connect("test.mosquitto.org", 1883, 60)
client.loop_start()
try:
# Simulate temperature readings
for i in range(5):
temperature = round(20 + random.uniform(-5, 10), 1)
message = f"{temperature}C"
result = client.publish("iotclass/lab1/temperature", message, qos=1)
result.wait_for_publish()
if result.rc != mqtt.MQTT_ERR_SUCCESS:
raise RuntimeError(f"Publish failed with MQTT code {result.rc}")
print(f"Published: {message}")
time.sleep(2)
finally:
client.loop_stop()
client.disconnect()
print("Done!")
With the subscriber running in one terminal, run the publisher in another:
python mqtt_publisher.py
In the offline control-flow test, five successful stubbed publish exchanges produced five synthetic Published: readings and requested five two-second sleeps. For a live trial, compare an actual Published: payload with a Received: line on your recorded topic. The code prints “Done!” after the cleanup calls in finally, including when a publish error exits the loop early; that line does not prove five successful exchanges.
If publication is reported but your subscriber stays empty, compare broker address and exact topics before changing timeouts. Check the subscriber’s connection result and running loop. Save both outputs so another person can follow the same reading.
Step 4: Experiment (10 min)
Make a prediction for each change. Attempt it before opening the feedback.
- Change the topic: Publish to
iotclass/lab1/yourname/temperaturewhile the subscriber keeps its original exact topic. Predict whether your new publication matches. Then restore the same distinctive topic in both programs and retry. - Inspect QoS: The publisher already has
qos=1; the check asks about its at-least-once exchange and possible duplicates. Answer its delivery question for QoS 1, then explain what changes when the receiver is offline. Keep the subscriber’s requested QoS separate from the publisher’s setting. - Use a JSON payload: Send the synthetic text
{"temp": 25.5, "unit": "C"}. The subscriber prints decoded text; that print alone does not validate JSON fields. - Try an offline receiver: Stop your subscriber, run your publisher, restart the subscriber, and publish again. Predict which readings you can prove arrived. Keep the offline trial separate from the wrong-topic trial.
Worked feedback: diagnose, restore, and retry
An exact subscription to the old topic does not match the added yourname level. Restoring the same topic in both programs removes that mismatch; it does not establish connectivity, authorization, or receipt. A bigger timeout cannot fix a topic mismatch.
The default subscriber has no explicit persistent-session setup, and the publisher does not request retained delivery. Do not assume readings sent while the subscriber was absent will replay on restart (MQTT 3.1.1 §§3.1.2.4 and 3.3.1.3; Paho client defaults). Publish again after the receiver connects and look for the new received payload.
QoS 1 can repeat a transfer when an acknowledgement is lost. A run with no visible duplicate does not show that duplicates are impossible. Applications must detect or tolerate repeated readings where repetition matters. The sender’s publish result and the subscriber’s output remain separate evidence.
Lab Validation
Compare your predictions with the offline control flow. A live trial would additionally need the actual Python and Paho versions, broker version, test date, topic, both outputs, and any correction followed by observed receipt. Stop only programs you started. The publisher calls loop_stop() and disconnect() in its finally block; the offline tests exercise these calls after success and injected publish/wait errors, not real socket cleanup. If running a subscriber, interrupt your own waiting process with Ctrl+C when finished.
Use this list for a live test. These are results you must collect, not results proved here:
- Record the connection result for the test broker.
- Check that the send topic and receive filter match.
- Match a received payload to one in your sender log.
- Explain the wrong-topic test. Restore the topic and check receipt again.
- In the offline test, record what arrived. Do not assume a replay.
- Record versions and the test date. Stop your own test programs and record the cleanup.
Summary
Compare the evidence you collected for the same missing reading:
- Retry schedule: Separate each delay from the sum of waits and from connection-attempt time.
- Retry state: Trace where the counter resets and what happens when one call exhausts its attempts.
- Callbacks and loop: Register handlers before starting service; keep the network loop running.
- Route and receipt: Compare exact topics, subscription state, and an actual received payload.
- Protection and timing: Keep TLS while measuring the connection path and configuring supported timeouts.
- Model and measurement: Label assumed constants and test capacity, current, and timing on the chosen system.
Knowledge Check
Attempt the final scenarios without reopening the worked feedback. For each choice, name the missing observation and explain why the other proposed changes would not address it. Then compare your reasoning with the check’s explanation.
Quiz: MQTT Python Patterns
Choose an answer for each stated scenario before reading its explanation. Name the observation that would support your diagnosis. Then open the scope notes after both questions.
Worked feedback: check the limits of each diagnosis
For the callback question, the scenario explicitly has no generic or topic-specific handler. Registering before the network thread starts avoids missing early messages; later assignment does not permanently disable a handler. A topic-specific callback can also handle a message instead of generic on_message, as the code challenge demonstrated.
For the TLS question, the trace supplies the missing cause evidence. Arduino ESP32 core 3.3.2 separates its TLS-handshake limit from MQTT keepalive and socket waiting. A timeout symptom alone does not establish the same remedy.
Concept Relationships
When a prediction fails, choose a guide to the part you need to test:
- MQTT Publisher-Subscriber Setup: build and watch the smallest message loop.
- MQTT Security: check how the link is protected, who the device is, and which topics it may use.
- MQTT QoS Levels: trace each delivery leg, from sender to broker and from broker to receiver.
- MQTT Production Operations: test how much load the broker can take. Track clients as they connect and drop off.
Retry rules work alongside routing, loop service, and the app’s message handling. If you change the retry rules, you still need to test those other parts.
