This lab implements advanced CoAP features on ESP32: the Observe pattern for push notifications when sensor values change (eliminating polling), block-wise transfer for large payloads like firmware updates across constrained networks, /.well-known/core resource discovery per RFC 6690, and CON message retransmission with exponential backoff for reliability over lossy links.
54.1 Learning Objectives
By the end of this chapter, you will be able to:
Implement CoAP Observe Pattern: Build servers that notify clients only when resource state changes
Use Block-wise Transfer: Handle large payloads (firmware updates) across constrained networks with small MTUs
Enable Resource Discovery: Expose /.well-known/core for automated service discovery per RFC 6690
Handle Retransmission: Implement CON message timeouts and exponential backoff
Assess DTLS Integration: Evaluate security layer requirements and select appropriate authentication modes (PSK vs certificates) for production CoAP deployments
Key Concepts
CoAP: Constrained Application Protocol — REST-style request/response protocol using UDP instead of TCP
Confirmable Message (CON): Requires ACK from recipient — provides reliable delivery over UDP at the cost of one roundtrip
Observe Option: CoAP extension enabling publish/subscribe: client registers to receive notifications on resource changes
Block-wise Transfer: Fragmentation mechanism for transferring payloads larger than a single CoAP datagram
Token: Client-generated value matching responses to requests — enables concurrent request/response pairing
DTLS: Datagram TLS — CoAP’s security layer providing encryption and authentication over UDP
54.2 For Beginners: CoAP Advanced Lab
This lab lets you experiment with advanced CoAP features like block transfers (sending large files in chunks) and observe notifications (getting automatic updates when a sensor value changes). Think of it as upgrading from basic text messaging to a full communication suite with file sharing and live updates.
54.3 Prerequisites
Before diving into this chapter, you should be familiar with:
Objective: Build a production-grade CoAP server on ESP32 demonstrating observe pattern, block-wise transfer, resource discovery, retransmission handling, and DTLS concepts.
What You’ll Learn:
CoAP Observe Pattern - Implement publish-subscribe with automatic notifications
Block-wise Transfer - Handle large payloads (firmware updates) across constrained networks
Resource Discovery - Expose /.well-known/core for service discovery
Retransmission Logic - Implement CON message timeouts and exponential backoff
Figure 54.1: Diagram showing CoAP server architecture with ESP32 connecting to temperature sensor, LED, and button, handling observe notifications, block transfers, and resource discovery
54.4.3 Interactive Wokwi Simulation
How to Use:
Click Start Simulation below
Open Serial Monitor to see CoAP server logs
Observe automatic notifications when temperature changes
Watch block-wise transfer simulation for large payloads
See resource discovery via /.well-known/core
Monitor CON message retransmissions on packet loss
{"version":1,"author":"CoAP Advanced Features Lab","editor":"wokwi","parts":[{"type":"wokwi-esp32-devkit-v1","id":"esp","top":0,"left":0,"attrs":{}},{"type":"wokwi-dht22","id":"dht","top":-10,"left":100,"attrs":{}},{"type":"wokwi-led","id":"led","top":50,"left":150,"attrs":{"color":"green"}},{"type":"wokwi-pushbutton","id":"btn","top":100,"left":100,"attrs":{"color":"blue"}}],"connections":[["esp:GND.1","dht:GND","black",["v0"]],["esp:3V3","dht:VCC","red",["v0"]],["esp:D15","dht:SDA","green",["v0"]],["esp:D2","led:A","yellow",["v0"]],["led:C","esp:GND.2","black",["v0"]],["btn:1.l","esp:D4","blue",["v0"]],["btn:1.r","esp:GND.3","black",["v0"]]]}
54.5 Interactive Calculators
Calculator: Block Transfer Time Estimator
Estimate transfer time for firmware updates using CoAP block-wise transfer.
Key Insight: Smaller block sizes reduce retry overhead on lossy networks but increase total block count. For 6LoWPAN networks, SZX=2 (64 bytes) typically provides the best balance.
Calculator: Observe Pattern Bandwidth Savings
Compare polling vs observe pattern for bandwidth efficiency.
Key Insight: Block sizes exceeding network MTU require fragmentation, causing exponential packet loss on unreliable wireless links. Always match block size to network constraints.
54.6 Challenge Exercises
Challenge 1: Implement Observe Deregistration
Goal: Handle GET /temperature with Observe: 1 (deregister) to stop notifications.
Steps:
In parsePacket(), detect when Observe option value is 1 (not 0)
In handleGetTemperature(), call removeObserver() when observeValue == 1
Verify with Serial Monitor that notifications stop after deregistration
Hint: The Observe option value is 0 for register, 1 for deregister.
Challenge 2: Add Retransmission Logic for CON Messages
Goal: Implement exponential backoff for CON messages that don’t receive ACK.
Steps:
Create a PendingMessage struct to track sent CON messages awaiting ACK
Start a timer (2s initial) when sending CON
If no ACK received, retransmit with doubled timeout per RFC 7252 (initial 2s, then retries: 4s → 8s → 16s → 32s)
After 4 retries, give up and increment stats.retransmissions
Hint: Use millis() for timing, store sent messages in an array.
Challenge 3: Optimize Block Size Negotiation
Goal: Respond with smaller block size if client requests too large.
Steps:
Parse client’s requested SZX from Block2 option
If server cannot support that size, respond with smaller SZX
Test with different SZX values (0=16B, 2=64B, 6=1024B)
Block2 encoding: NUM (12 bits) | M (1 bit) | SZX (3 bits)
Challenge 4: Add DTLS Pre-Shared Key Simulation
Goal: Simulate DTLS PSK validation before processing requests.
Steps:
Define a pre-shared key (e.g., const char* PSK = "secret123")
Add a simple header check (e.g., first 8 bytes of payload)
Reject requests without valid PSK with 4.01 Unauthorized
This is NOT real DTLS - just a simulation for understanding
Note: Real DTLS requires a proper library like mbed TLS.
Worked Example: CoAP Block Transfer for Firmware Update
Scenario: An agricultural sensor needs to receive a 48 KB firmware update over a 6LoWPAN network with 127-byte MTU (after 6LoWPAN headers, only 80 bytes available for CoAP payload).
Given:
Firmware size: 48 KB (49,152 bytes)
Available payload per block: 64 bytes (Block2 SZX=2, leaving room for CoAP header and options)
Network packet loss rate: 5%
Round-trip time (RTT): 200 ms
CoAP CON message timeout: 2 seconds with exponential backoff
Steps:
Calculate number of blocks required:
Total blocks = ceil(49,152 / 64) = 768 blocks
Block numbering: Block2 option uses NUM field (0-767)
Estimate transmission time (ideal, no loss):
Each block requires request + response = 1 RTT = 200 ms
Total time = 768 blocks × 200 ms = 153.6 seconds (2 min 34 sec)
Account for packet loss with retransmissions:
Expected lost blocks = 768 × 5% = ~38 blocks
First retry adds 2 sec timeout + 200 ms RTT = 2.2 sec each
Additional time = 38 × 2.2 sec = 83.6 seconds
Total estimated time = 153.6 + 83.6 = 237.2 seconds (3 min 57 sec)
Verify Block2 option encoding:
Block2 format: NUM (variable) | M (1 bit) | SZX (3 bits)
Result: The 48 KB firmware transfers in approximately 4 minutes over the constrained 6LoWPAN link, using 768 individually acknowledged 64-byte blocks. The client can resume from any failed block without restarting the entire transfer.
Key Insight: Block transfer enables reliable large payload transmission over constrained networks by trading latency for reliability. The ability to resume from any block position makes it ideal for unreliable networks where connection drops are common. Choose block size (SZX) based on network MTU: SZX=2 (64B) for 6LoWPAN, SZX=6 (1024B) for Wi-Fi.
Putting Numbers to It
When calculating optimal block size for constrained networks, the efficiency depends on the ratio of payload to overhead.
Check Your Understanding: Block Transfer Efficiency
54.10 Common Pitfall: Block Transfer SZX Negotiation
Pitfall: Ignoring Block Transfer SZX Negotiation
The Mistake: Developers hardcode Block2 SZX (size exponent) to the maximum value (SZX=6 for 1024-byte blocks) assuming larger blocks are always more efficient, causing firmware updates to fail on 6LoWPAN or Thread networks.
Why It Happens: The misconception that “fewer blocks = faster transfer” ignores network MTU constraints. IEEE 802.15.4 networks have 127-byte physical layer MTU, which after 6LoWPAN headers leaves only 80-100 bytes for CoAP payload. A 1024-byte block request triggers IP fragmentation, causing packet loss rates of 30-50% on lossy wireless links.
The Fix: Always negotiate block size based on network constraints using the Block2 option’s SZX field (size = 2^(SZX+4) bytes):
SZX=0 (16 bytes): Ultra-constrained networks, LoRa with small payloads
Implement adaptive block sizing: start with SZX=6, and if the server responds with a smaller SZX in the Block2 option, reduce to match. Example: client requests Block2 SZX=6, server responds with Block2 SZX=2 indicating its MTU constraint.
54.11 Visual Reference Gallery
Visual: CoAP Message Types
Figure 54.2: CoAP Message Types showing CON, NON, ACK, RST
Understanding CoAP’s four message types is essential for designing efficient IoT communication patterns that balance reliability requirements against power and bandwidth constraints.
Visual: CoAP Observe Pattern
Figure 54.3: CoAP Observe Pattern for push notifications
The Observe extension enables efficient publish-subscribe semantics over CoAP’s request-response model, achieving significant power savings compared to traditional polling approaches.
Visual: CoAP Block Transfer
Figure 54.4: CoAP Block Transfer for large payloads
Block transfer is critical for practical IoT applications requiring firmware updates or bulk data transfer over networks with small MTU sizes and limited buffer capacity.
Interactive: CoAP Congestion Control Animation
Label the Diagram
💻 Code Challenge
Order the Steps
Match the Concepts
54.12 Summary
This advanced lab demonstrated production-grade CoAP features on ESP32:
Observe Pattern: Implemented push notifications with sequence numbers, reducing polling by 97%+ for slowly-changing data
Block-wise Transfer: Handled 4KB firmware in 64-byte blocks, enabling resumable transfers on constrained networks
Resource Discovery: Exposed /.well-known/core in RFC 6690 Link Format for automated service discovery
Server Architecture: Built multi-resource CoAP server with temperature sensor, configuration, and firmware endpoints
1. Using Confirmable Messages Everywhere for “Safety”
CoAP Confirmable (CON) messages add ACK overhead that can overwhelm constrained networks. For high-frequency sensor readings, Non-confirmable (NON) messages reduce overhead by 40-60%. Reserve CON only for actuator commands and critical alerts.
2. Ignoring Block-Wise Transfer for Large Payloads
Sending payloads larger than the network MTU without Block-Wise (RFC 7959) causes fragmentation at lower layers, degrading reliability. Any payload exceeding ~1KB should use Block2/Block1 options to transfer in 512-1024 byte chunks.
3. Not Handling Observe Notification Timeouts
CoAP Observe registrations expire silently if the server doesn’t receive a CON notification ACK. Implement registration refresh every 24 hours and handle the case where the observer re-registers after a network disruption.
54.16 What’s Next
Having implemented advanced CoAP features on ESP32, you are ready to consolidate your knowledge and explore related protocols and security layers.