1234  CoAP Review: Observe Patterns and Push Notifications

1234.1 Learning Objectives

By the end of this chapter, you will be able to:

  • Implement CoAP Observe: Set up server-push notifications for real-time sensor monitoring
  • Handle Sequence Numbers: Correctly process notification ordering and wraparound
  • Manage Subscriptions: Register, maintain, and deregister Observe relationships
  • Optimize Energy: Apply conditional observe to reduce unnecessary transmissions
  • Troubleshoot NAT Issues: Solve common firewall/NAT timeout problems

1234.2 Prerequisites

Required Reading:

Estimated Time: 25 minutes

This is Part 2 of the CoAP Comprehensive Review:

  1. Protocol Fundamentals - Message types, architecture, comparisons
  2. Observe Patterns (this chapter) - Push notifications, subscriptions
  3. Labs & Implementation - Hands-on ESP32 and Python
  4. Knowledge Assessment - Quizzes and scenario questions

1234.3 CoAP Observe Mechanism - Push Notifications

The Observe pattern is CoAP’s killer feature for IoT, enabling server-push notifications without persistent connections. This provides 99% energy savings compared to HTTP polling:

%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#2C3E50', 'primaryTextColor': '#fff', 'primaryBorderColor': '#16A085', 'lineColor': '#16A085', 'secondaryColor': '#E67E22', 'tertiaryColor': '#7F8C8D', 'fontSize': '14px'}}}%%
sequenceDiagram
    participant C as Client
    participant S as Server (Sensor)

    Note over C,S: Phase 1: Registration (3 mJ)
    C->>S: CON GET /temp<br/>Observe: 0 (register)
    S->>C: ACK 2.05 Content<br/>Observe: 42, temp=22C
    Note over C,S: Subscription established!

    Note over C,S: <br/>Phase 2: Notifications (2.5 mJ each)
    Note over S: Temperature changes to 23C
    S->>C: CON 2.05 Content<br/>Observe: 43, temp=23C
    C->>S: ACK [MID]

    Note over S: Temperature changes to 24C
    S->>C: CON 2.05 Content<br/>Observe: 44, temp=24C
    C->>S: ACK [MID]
    Note over C,S: Server pushes updates autonomously<br/>Zero client polling needed!

    Note over C,S: <br/>Phase 3: Deregistration
    C->>S: RST (stop observing)
    Note over S: Observation stopped

    Note over C,S: <br/>Alternative: NAT/Firewall Workaround
    Note over C: Re-register every 5 min<br/>to refresh UDP mapping
    C->>S: CON GET /temp<br/>Observe: 0 (refresh)
    S->>C: ACK with current value

Figure 1234.1: CoAP Observe server-push notifications with 99% energy savings

CoAP Observe mechanism demonstrating server-push notifications with 99% energy savings versus client polling. Phase 1 - Registration (one-time 3 mJ cost): Client sends single CON GET request with Observe: 0 option header to register interest in /temp resource. Server responds with ACK 2.05 Content containing initial temperature value (22C) and Observe sequence number 42. Subscription now established - client doesn’t send further requests. Phase 2 - Autonomous Notifications (2.5 mJ each): When sensor detects temperature change to 23C, server autonomously pushes CON notification with Observe: 43 and new value, without waiting for client request. Client sends ACK to confirm receipt. Server continues pushing notifications for each change (24C with Observe: 44), maintaining incrementing sequence numbers. Client compares sequence numbers to detect out-of-order delivery caused by network reordering - if received seq < last_seen_seq, discard as stale. Zero client polling overhead, zero network bandwidth waste, instant 0-latency updates when data changes. Phase 3 - Deregistration (clean termination): Client sends RST message in response to notification to explicitly stop subscription, or alternatively sends GET with Observe: 1 to formally deregister. Server removes observation from active list, stops sending notifications. Server also implements 24-hour timeout to automatically clean up stale observations if client disappears.

Figure 1234.2

1234.4 Observe Performance Analysis

Scenario: Temperature monitoring over 24 hours, sensor updates every 5 minutes (288 actual changes)

Approach Messages Energy Bandwidth Latency Notes
HTTP Polling (30s) 2,880 requests 86,400 mJ 464.1 KB 15s avg Client polls every 30s
CoAP NON Polling (30s) 2,880 requests 43,200 mJ 61.9 KB 15s avg CoAP without Observe
CoAP Observe 289 messages 723 mJ 7.4 KB 0s (instant) 1 register + 288 notifications
Conditional Observe 58 messages 145 mJ 1.5 KB 0s (instant) Only notify if change >0.5C

Key Benefits:

  1. Energy Efficiency: 99% reduction vs HTTP polling (723 mJ vs 86,400 mJ)
  2. Instant Updates: 0-second latency vs 15-second average staleness with polling
  3. Bandwidth Savings: 98.4% reduction (7.4 KB vs 464.1 KB)
  4. Server-Initiated: No client polling overhead - server pushes when data changes
  5. Scalability: 1 server can handle 1000+ concurrent observations efficiently

1234.5 Common Pitfalls and Solutions

1234.5.1 NAT/Firewall Timeout Issues

  • Problem: NAT/firewall closes UDP port after 60-120 seconds of inactivity
    • Solution: Client re-registers every 5 minutes to refresh port mapping
  • Problem: Notifications arrive out-of-order due to network reordering
    • Solution: Check Observe sequence numbers; discard if seq# < last_received
  • Problem: Server doesn’t know client lost power/restarted
    • Solution: Server implements 24-hour timeout; cleans up stale observations
  • Problem: Battery drains fast with frequent notifications
    • Solution: Use conditional observe (only notify on significant changes >0.5C)
CautionPitfall: Ignoring NAT/Firewall Timeouts with CoAP Observe

The Mistake: Developers implement CoAP Observe for real-time sensor monitoring but don’t account for NAT/firewall UDP session timeouts. After 60-300 seconds of inactivity, the NAT mapping expires and server notifications can no longer reach the client.

Why It Happens: Observe works perfectly in development (local network, no NAT), but fails in production when devices are behind home routers or enterprise firewalls. Developers assume the UDP “connection” stays open indefinitely like TCP.

The Fix: Implement Observe re-registration before NAT timeout:

# Re-register Observe every 5 minutes to refresh NAT mapping
OBSERVE_REFRESH_INTERVAL = 300  # seconds (5 minutes)

async def maintain_observe(uri):
    while True:
        # Send fresh GET with Observe:0 to re-register
        request = Message(code=GET, uri=uri)
        request.opt.observe = 0
        await protocol.request(request).response
        await asyncio.sleep(OBSERVE_REFRESH_INTERVAL)

Alternatively, use CON (Confirmable) notifications instead of NON - the bidirectional ACK traffic keeps NAT mappings alive.

Real Impact: A smart agriculture deployment lost 40% of sensor data because Observe notifications stopped arriving after NAT timeouts. The sensors were sending data, but packets were dropped at the farm’s router. Re-registration every 5 minutes restored 99.7% data delivery with minimal overhead (1 extra message every 5 minutes vs continuous polling).

CautionPitfall: Using NON Messages for Critical Commands

The Mistake: Developers use Non-Confirmable (NON) messages for actuator commands like door locks, HVAC controls, or emergency stops because “CoAP is reliable enough” or to save energy.

Why It Happens: NON messages are simpler (no ACK handling) and use 50% less energy than CON. In testing with reliable networks, NON appears to work fine. The 5-15% packet loss in real wireless environments isn’t apparent until production.

The Fix: Always use CON (Confirmable) messages for commands that affect physical state:

# Bad: NON for actuator command
request = Message(code=PUT, uri='coap://lock/state', mtype=NON)
request.payload = b'{"locked": false}'
await protocol.request(request).response  # May never arrive!

# Good: CON for actuator command with retry
request = Message(code=PUT, uri='coap://lock/state', mtype=CON)
request.payload = b'{"locked": false}'
response = await protocol.request(request).response  # Retries automatically
if response.code == CHANGED:
    print("Door unlocked successfully")

Real Impact: A warehouse automation system used NON messages for forklift routing commands. During Wi-Fi congestion, 8% of commands were lost, causing forklifts to collide. After switching to CON messages, command delivery reached 99.99% with automatic retransmission. The 2x energy cost was negligible compared to collision damage costs. Rule of thumb: NON for sensors (temperature, humidity), CON for actuators (locks, motors, valves).

1234.6 Worked Example: Observe Token and Sequence Number Management

NoteWorked Example: Observe Token and Sequence Number Management

Scenario: A CoAP client monitors a temperature sensor using the Observe extension. You need to implement proper token matching and sequence number validation to handle out-of-order notifications and detect stale data.

Given:

  • Client token for observation: 0xA7B3 (2 bytes)
  • Initial response Observe sequence: 1000
  • Subsequent notifications arrive: seq=1002, seq=1001, seq=1003, seq=1005
  • Max Observe value: 2^24 - 1 = 16,777,215 (3-byte encoding)
  • Network reordering window: up to 128 seconds

Steps:

  1. Parse initial registration response:

    Response: 2.05 Content
Token: 0xA7B3 (matches request - accept)
Observe: 1000 (initial sequence)
Payload: {"temp": 22.5, "unit": "C"}

Client state:
- last_seq = 1000
- last_time = T0 (current timestamp)
- registered = true

  2. Process notification seq=1002:

    • Token matches: 0xA7B3 = 0xA7B3 (accept)
    • Sequence check: 1002 > 1000 (newer, accept)
    • Update state: last_seq = 1002, last_time = T1
    • Result: Accept notification

  3. Process notification seq=1001 (out-of-order):

    • Token matches: accept
    • Sequence check: 1001 < 1002 (older)
    • Time check: |T2 - T1| < 128 seconds (within window)
    • Fresher data already received
    • Result: Discard as stale

  4. Process notification seq=1003:

    • Token matches: accept
    • Sequence: 1003 > 1002 (newer)
    • Result: Accept, update last_seq = 1003

  5. Handle sequence wraparound (hypothetical: current=16,777,200, new=5):

    def is_fresh(new_seq, last_seq):
    MAX_SEQ = 2**24
    HALF_MAX = 2**23

    diff = (new_seq - last_seq) % MAX_SEQ

    # If diff < 2^23, new_seq is "ahead" (accounting for wrap)
    if diff < HALF_MAX:
        return True  # Fresh notification
    else:
        return False  # Stale (arrived late)

# Example: new=5, last=16,777,200
# diff = (5 - 16,777,200) % 16,777,216 = 21
# 21 < 8,388,608, so fresh = True (wrapped around)

Result: The client correctly processes notifications 1000, 1002, 1003, and 1005, while discarding the late-arriving 1001. The sequence number algorithm handles both normal ordering and 24-bit wraparound scenarios.

Key Insight: CoAP Observe uses a 24-bit sequence space that wraps around. Clients must implement modular arithmetic comparison (not simple <> comparison) to correctly detect fresh vs stale notifications. Always validate token first (security), then sequence number (freshness). Discard notifications with mismatched tokens - they may be spoofed or from a different observation.

1234.7 Understanding Check: Observe Patterns

Situation: Your smart building has three types of devices with different reliability requirements:

Device types:

  1. Temperature sensors: Report every 30 seconds
  2. Fire alarms: Critical alerts that MUST be received
  3. Occupancy counters: Periodic counts where occasional loss acceptable

Think about:

  1. What happens if a temperature sensor’s NON message is lost?
  2. What happens if a fire alarm’s NON message is lost?
  3. How does CON message reliability work without TCP?

Key Insight:

NON (Non-Confirmable):

  • Fire-and-forget: Send once, no acknowledgment
  • Energy: 1.5 mJ (transmit only, 25ms)
  • Reliability: ~85-95% (depends on network quality)
  • Use for: Replaceable data where next update supersedes previous

CON (Confirmable):

  • Wait for ACK: Server acknowledges receipt
  • Energy: 3.0 mJ (transmit + wait for ACK, 50ms)
  • Reliability: 99.99% (retries up to 4 times with exponential backoff)
  • Use for: Critical data that MUST arrive

Recommended mapping:

Temperature sensors -> NON (missing 1 reading OK, next arrives in 30s)
Fire alarms -> CON (MUST deliver, worth 2x energy cost)
Occupancy -> NON (periodic count, loss has minimal impact)

Energy impact over 24 hours:

  • Temp sensor (2880 readings/day): NON = 4.3Wh, CON = 8.6Wh (2x difference)
  • Fire alarm (2 alarms/year): Energy impact negligible, reliability critical
  • Occupancy (96 counts/day): NON appropriate

Situation: You’re choosing a protocol for a warehouse automation system with 200 temperature sensors and 50 automated forklifts.

Requirements:

  • Sensors: Report temperature to central monitoring (one-to-one)
  • Forklifts: Send location updates to multiple subscribers (dispatch, safety, analytics)

Think about:

  1. Which communication pattern does each device type need?
  2. Does CoAP support publish-subscribe like MQTT?
  3. How would you implement CoAP for the sensors vs MQTT for the forklifts?

Key Insight:

Architectural patterns:

CoAP (Request-Response):

Client -> Server pattern
Sensor: POST coap://server/warehouse/zone1/temp {"value": 24.5}
Server: ACK 2.04 Changed
  • Direct device-to-server communication
  • RESTful API (GET /temp, POST /temp)
  • No broker infrastructure needed
  • One publisher -> One subscriber

MQTT (Publish-Subscribe):

Publisher -> Broker -> Subscribers pattern
Forklift: PUBLISH forklift/42/location {"x": 125, "y": 340}
Broker: Routes to ALL subscribers of forklift/42/location
Subscribers: Dispatch system, safety monitor, analytics
  • Decoupled: Publishers don’t know subscribers
  • One publisher -> Many subscribers
  • Broker required (single point of management/failure)

Recommended architecture:

Temperature sensors (200) -> CoAP -> Central server
- Direct communication, no broker overhead
- RESTful endpoints: coap://server/temp/zone1, coap://server/temp/zone2
- Lower latency (no broker hop)
- Simpler infrastructure

Forklifts (50) -> MQTT -> Broker -> Multiple apps
- Dispatch needs location updates
- Safety system needs location updates
- Analytics needs location history
- MQTT naturally broadcasts to all subscribers

Hybrid benefits:

  • CoAP for one-to-one (sensors): Minimal overhead, RESTful API
  • MQTT for one-to-many (forklifts): Natural pub-sub pattern
  • Total cost: 1 MQTT broker + 1 CoAP server (vs 250 devices x broker overhead)

1234.8 Visual Reference Gallery

Modern visualization of CoAP Observe pattern showing client registration, server state changes triggering notifications, and efficient push-based updates replacing inefficient polling

CoAP Observe showing push notifications

The Observe pattern is CoAP’s key feature for efficient IoT monitoring, enabling 99% bandwidth reduction compared to polling by only sending updates when sensor values actually change.

1234.9 Summary

This chapter covered CoAP Observe patterns for efficient push notifications:

  • Registration: Single GET with Observe: 0 establishes subscription
  • Notifications: Server autonomously pushes updates when data changes
  • Sequence Numbers: 24-bit counter with wraparound handling for freshness detection
  • Deregistration: RST message or GET with Observe: 1 stops notifications
  • Energy Savings: 99% reduction vs polling (723 mJ vs 86,400 mJ over 24 hours)
  • NAT Workarounds: Re-register every 5 minutes to refresh UDP port mappings

1234.10 What’s Next

Continue to CoAP Review: Labs & Implementation for hands-on experience building CoAP servers on ESP32 and Python with sensors and actuators.