14 CoAP Observe: Efficiency and Lifecycle

coap

protocols

observe

server-push

14.1 Start With the Decision

A NAT may forget an idle observer before the next sensor change. Keep-alive cost must be weighed against a new connection path.

14.2 Route Overview

This is part 2 of 2. Review CoAP Observe: Subscriptions and Updates for the preceding evidence.

14.3 Learning Objectives

Calculate a NAT keep-alive interval and its traffic.
Compare Observe traffic, battery wakes, and client restarts.

14.4 Chapter Roadmap

Interactive Calculator: NAT Keep-Alive Interval
Pitfall: Mismanaging Observe Tokens Across Client Restarts
Bandwidth Savings Calculation
Interactive Calculator: Battery Life Comparison
Interactive: CoAP Observe Pattern Animation
Worked Example: Industrial Vibration Monitoring with Observe
Decision Framework: When to Use CoAP Observe
Common Mistake: Forgetting to Handle Observer Cleanup After Client Crashes
Checkpoint: Failure Modes
Common Pitfalls
1. Using Confirmable Messages for Every CoAP Request
2. Ignoring CoAP Proxy Caching Semantics
3. Forgetting DTLS Session Management
Label the Diagram
Order the Steps
Practice Exercises
Hands-On Practice
Checkpoint: Evidence Before Selection
Concept Relationships
See Also
Match the Concepts
What’s Next
Summary

Interactive Calculator: NAT Keep-Alive Interval

How to use: Enter your NAT timeout and number of observers. The calculator recommends a keep-alive interval with safety margin and shows the bandwidth overhead. Use this to decide if keep-alives are practical or if you should switch to CoAP over TCP.

Pitfall: Mismanaging Observe Tokens Across Client Restarts

The Mistake: Clients generate new random tokens after reboot without deregistering previous observations, causing “ghost subscriptions” where the server continues sending notifications to tokens the client no longer recognizes.

Why It Happens: The Observe pattern uses tokens to match notifications to subscriptions. When a client reboots, it loses its token-to-subscription mapping but the server still has the observer registered.

The Fix: Implement proper token lifecycle management:

Persist tokens across reboots: Store active observation tokens in EEPROM/Flash
Use deterministic token generation: Generate from device ID + resource URI hash
Handle orphaned notifications gracefully: When receiving notification with unknown token, send RST
Server-side timeout: Configure observer timeout (Max-Age option)

14.5 Bandwidth Savings Calculation

Example: Temperature sensor with 100 observers

Polling approach (GET every 10 seconds): Read these points as one connected sequence: start with Request: 14 bytes (header + token + Uri-Path); then Response: 16 bytes (header + token + payload); and finish with Total: 30 bytes x 100 clients x 6/min x 60 min = 1.08 MB/hour.

Request: 14 bytes (header + token + Uri-Path)
Response: 16 bytes (header + token + payload)
Total: 30 bytes x 100 clients x 6/min x 60 min = 1.08 MB/hour

Observe approach (notify on change, avg 6 changes/hour): Read these points as one connected sequence: start with CoAP header: 4 bytes; then Token: 2 bytes; then Observe option: 3 bytes; then Content-Format: 2 bytes; then Payload marker: 1 byte; then Payload: 6 bytes ("22.5"); and finish with Total: 18 bytes x 100 clients x 6 changes = 10.8 KB/hour.

CoAP header: 4 bytes
Token: 2 bytes
Observe option: 3 bytes
Content-Format: 2 bytes
Payload marker: 1 byte
Payload: 6 bytes ("22.5")
Total: 18 bytes x 100 clients x 6 changes = 10.8 KB/hour

Savings: 99% bandwidth reduction (1.08 MB vs 10.8 kB)

Interactive Calculator: Battery Life Comparison

How to use: Enter your battery specifications and usage patterns. The calculator shows battery life for both polling and Observe approaches, plus the improvement factor. This helps justify the engineering effort of implementing Observe.

Interactive: CoAP Observe Pattern Animation

Worked Example: Industrial Vibration Monitoring with Observe

Scenario: A manufacturing plant monitors vibration levels on 100 motors using ESP32 sensors with accelerometers. Maintenance dashboard needs real-time updates when vibration exceeds thresholds (normal: <2.5 mm/s, warning: 2.5-4.5 mm/s, critical: >4.5 mm/s).

Comparing polling vs CoAP Observe:

Option A: Polling (GET every 5 seconds):

Client polls: GET coap://motor42.local/vibration every 5 seconds
  Request: 16 bytes (CoAP header + token + URI)
  Response: 20 bytes (CoAP header + token + payload "2.3")

Per sensor traffic (24 hours):
  17,280 requests × 16 bytes = 276,480 bytes
  17,280 responses × 20 bytes = 345,600 bytes
  Total: 622,080 bytes/day per sensor

Fleet traffic: 622,080 × 100 = 62.2 MB/day
Energy per sensor: 17,280 × 3.0 mJ (CON request-response) = 51.8 J/day
Battery life (18650, 3.7V, 3,000 mAh): 40 kJ ÷ 51.8 J/day = ~772 days

Option B: CoAP Observe (notify on change, max 1/minute):

Client registers: GET /vibration, Observe: 0 (once at startup)
  Registration: 18 bytes (request) + 22 bytes (response with Observe: 1)

Server notifies only when vibration crosses thresholds:
  Typical motor: 3 threshold crossings/day (normal ↔ warning ↔ critical)
  Notifications: 3 × 20 bytes = 60 bytes/day
  Plus max-rate limit: If vibrating continuously, max 1,440 notifications/day

Per sensor traffic (stable operation, 3 changes/day):
  Registration: 40 bytes (one-time)
  Notifications: 60 bytes/day
  Total: ~100 bytes/day

Fleet traffic: 100 × 100 sensors = 10 KB/day (vs 62.2 MB with polling)
Energy: 3 notifications × 1.5 mJ = 4.5 mJ/day
Battery life: 40 kJ ÷ 0.0045 kJ/day = ~24,000 days (limited by battery shelf life)

Bandwidth savings: (62.2 MB - 0.01 MB) / 62.2 MB = 99.98%
Battery life improvement: 24,000 / 772 = 31× longer

Implementation with rate limiting:

class RateLimitedVibrationResource:
    def __init__(self):
        self.min_notify_interval = 60  # seconds (max 1/minute)
        self.threshold_change = 0.5     # mm/s (notify if change ≥ 0.5)
        self.last_notify_time = {}
        self.last_notified_value = {}

    async def notify_observers(self, new_value):
        now = time.time()
        for observer in self.observers:
            last_time = self.last_notify_time.get(observer.token, 0)
            last_value = self.last_notified_value.get(observer.token, None)

            # Notify if: threshold crossed OR min interval passed
            threshold_crossed = (
                last_value is not None and
                abs(new_value - last_value) >= self.threshold_change
            )
            interval_passed = (now - last_time) >= self.min_notify_interval

            if threshold_crossed or interval_passed:
                await self.send_notification(observer, new_value)
                self.last_notify_time[observer.token] = now
                self.last_notified_value[observer.token] = new_value

Decision: CoAP Observe

Reasoning:

99.98% bandwidth reduction (critical for cellular backhaul)
31× battery life extension (2 years → 65 years, limited by battery shelf life)
Real-time alerts (no polling delay)
Server-side rate limiting prevents notification floods

Decision Framework: When to Use CoAP Observe

Use this flowchart to decide if Observe is appropriate for your application:

Requirement	Use Observe	Use Polling	Use Neither (MQTT)
Update frequency	Event-driven, infrequent changes	Periodic, consistent intervals	Continuous streaming
Number of observers	1-10 per resource	1-3 per resource	10+ observers (broker scales better)
Change rate	<10% of polling rate	>50% of polling rate	Constant changes
Battery constraints	Critical (coin cell, multi-year)	Moderate (rechargeable)	Mains-powered
Network reliability	Stable (LAN, Wi-Fi)	Lossy (cellular)	Stable with fallback
Observer lifecycle	Long-lived (hours to days)	Short-lived (seconds to minutes)	Permanent subscriptions

Decision tree:

Does the resource change less often than you would poll it? → No: Use polling (Observe overhead not justified) → Yes: Continue
Do you need notifications from more than 10 resources per client? → Yes: Consider MQTT (broker-based pub-sub scales better) → No: Continue
Are clients behind NAT/firewall without port forwarding? → Yes: Problem - server can’t push to client. Options:
- CoAP over TCP (RFC 8323) for NAT traversal
- Long polling instead of Observe
- MQTT instead → No: Continue
Is battery life critical (>1 year target on coin cell)? → Yes: Use CoAP Observe (eliminate polling overhead) → No: Polling is acceptable, but Observe still beneficial

Example decisions:

Application	Observe?	Reasoning
Temperature sensor (changes every 30 min)	Yes	Polling every 5 min wastes 5× bandwidth
Stock price ticker (changes every second)	No	Polling every second = always changing, no savings
Door sensor (changes 10×/day)	Yes	Massive savings (99% fewer messages)
Accelerometer (100 Hz continuous)	No	Use MQTT or streaming protocol
Smart meter (reads every 15 min)	Depends	If value changes every time, polling OK. If often unchanged, Observe better

Common Mistake: Forgetting to Handle Observer Cleanup After Client Crashes

The Error: Server registers Observe subscriptions but never removes them when clients crash or restart, leading to “ghost observers” that consume memory and network bandwidth sending notifications to unreachable clients.

Why It Happens: CoAP Observe uses tokens to match notifications to requests. When a client crashes and restarts, it generates a new random token. The server continues sending notifications to the old token, which are now unrecognized and trigger RST responses.

Real-World Impact: An industrial monitoring system with 500 sensors and 20 dashboard clients (web browsers):

Without observer cleanup:

Each browser refresh creates new Observe subscription (new token)
Each old subscription remains active (server doesn't know browser closed)

After 1 week (browsers refresh ~50 times each):
  Ghost observers: 20 clients × 50 refreshes = 1,000 stale subscriptions
  Active observers: 20 clients = 20 valid subscriptions
  Ratio: 1,000 / 20 = 50:1 ghost to valid

Notification overhead:
  500 sensors × 10 changes/hour × 1,020 observers = 5.1M notifications/hour
  Wasted: 5M going to ghost observers (98% waste)
  Server CPU: 45% spent serializing notifications for dead clients
  Network: 850 MB/day of wasted traffic (RST responses)

The Fix:

1. Max-Age timeout (automatic cleanup):

# Server sets observation lifetime
response.opt.max_age = 3600  # Expire after 1 hour

# Observer must re-register before expiration or be removed
def cleanup_expired_observers(self):
    now = time.time()
    for resource_uri, observers in self.observers.items():
        self.observers[resource_uri] = [
            o for o in observers
            if now - o.registered_at < o.timeout
        ]

2. RST detection (immediate cleanup):

def on_rst_received(self, client_addr, message_id):
    """Remove observer when client sends RST to notification"""
    for resource_uri, observers in self.observers.items():
        self.observers[resource_uri] = [
            o for o in observers
            if not (o.client_addr == client_addr and o.last_mid == message_id)
        ]
    logging.info(f"Removed observer {client_addr} after RST")

3. CON notification failures (retry limit):

async def send_notification(self, observer, value):
    """Send CON notification, remove observer after 4 failed retries"""
    msg = Message(code=CONTENT, token=observer.token, payload=value)
    msg.mtype = CON  # Confirmable - requires ACK

    for attempt in range(4):
        try:
            await self.send_message(observer.client_addr, msg)
            ack = await self.wait_for_ack(msg.mid, timeout=2 * (2 ** attempt))
            return  # Success
        except TimeoutError:
            logging.warning(f"Notification attempt {attempt+1} failed for {observer}")

    # 4 failures - remove observer
    self.remove_observer(resource_uri, observer)
    logging.info(f"Removed unresponsive observer {observer.client_addr}")

Results after implementing cleanup:

Ghost observers after 1 week: 0 (all cleaned up within 1 hour)
Notification waste: 0% (only sending to active clients)
Server CPU: 5% (down from 45%)
Network traffic: 8.5 MB/day (down from 858 MB/day)

Prevention checklist:

Set Max-Age on all Observe responses (default: 1 hour)
Implement RST detection and immediate observer removal
Use CON notifications (not NON) so you detect unreachable clients
Limit retries to 4 attempts (RFC 7252 default) before removal
Log observer registration/removal for debugging
Monitor observer count per resource (alert if >expected)

Checkpoint: Failure Modes

You now know unknown tokens after a restart should trigger RST cleanup rather than silent drops.
You now know UDP NAT mappings can expire after 30-60 seconds, so quiet Observe streams may need keep-alives.
You now know Max-Age, RST handling, and CON retry failure are three cleanup tools for ghost observers.

Common Pitfalls

1. Using Confirmable Messages for Every CoAP Request

CON messages require an ACK roundtrip — on lossy networks with 20% packet loss, a 4-attempt retry with exponential backoff can delay responses by 45 seconds. Use NON for periodic telemetry where data freshness matters more than guaranteed delivery; reserve CON for actuation commands.

2. Ignoring CoAP Proxy Caching Semantics

CoAP proxies cache GET responses based on Max-Age option — a sensor returning temperature with Max-Age=60 will serve cached values for 60 seconds even if the physical reading changes. Set Max-Age to match your data freshness requirement, not the default 60 seconds.

3. Forgetting DTLS Session Management

DTLS handshake (6-8 roundtrips) dominates latency for short-lived CoAP connections — repeatedly creating new DTLS sessions for each request adds 500-2000 ms overhead. Use DTLS session resumption (RFC 5077) to reduce reconnection to 1 roundtrip after the initial handshake.

Label the Diagram

Order the Steps

14.6 Practice Exercises

Hands-On Practice

Exercise 1: Observer Registry Design Design an observer registry that supports: Read these points as one connected sequence: start with Maximum 50 observers per resource; then Automatic cleanup of stale observers (> 24 hours); and finish with Rate limiting: max 1 notification per second per observer.

Maximum 50 observers per resource
Automatic cleanup of stale observers (> 24 hours)
Rate limiting: max 1 notification per second per observer

Exercise 2: Bandwidth Calculation A smart building has 200 temperature sensors, each observed by 3 clients (dashboard, HVAC controller, alarm system). Sensors report every time temperature changes by 0.5°C. Calculate: Read these points as one connected sequence: start with Estimated notifications per hour (assuming 2 significant changes per sensor per hour); then Bandwidth usage with 20-byte notification payloads; and finish with Savings compared to polling every 30 seconds.

Estimated notifications per hour (assuming 2 significant changes per sensor per hour)
Bandwidth usage with 20-byte notification payloads
Savings compared to polling every 30 seconds

Exercise 3: NAT Keep-Alive Strategy Design a keep-alive strategy for a CoAP server where: Read these points as one connected sequence: start with NAT timeout is 45 seconds; then Server has 1,000 active observers; and finish with Network bandwidth is limited to 10 Kbps for keep-alives.

NAT timeout is 45 seconds
Server has 1,000 active observers
Network bandwidth is limited to 10 Kbps for keep-alives

What keep-alive interval would you choose and why?

Checkpoint: Evidence Before Selection

You now know the chapter’s 100-observer example can show 99% bandwidth reduction when changes are infrequent.
You now know the vibration example compares about 772 days of polling battery life with a much longer Observe lifetime.
You now know MQTT becomes the better comparison when subscriber counts or continuous streams outgrow direct server push.

14.7 Concept Relationships

How CoAP Observe connects to broader IoT patterns and protocols:

Observe builds on:

Read these points as one connected sequence: start with CoAP Message Types - Uses CON/NON for notifications with sequence number tracking; and finish with UDP Transport - Stateless protocol requiring application-layer state management.

CoAP Message Types - Uses CON/NON for notifications with sequence number tracking
UDP Transport - Stateless protocol requiring application-layer state management

Similar patterns in other protocols:

Read these points as one connected sequence: start with MQTT Subscriptions - Topic-based pub/sub vs URI-based observe; then WebSocket Server Push - Full-duplex vs observe’s asymmetric push; and finish with Server-Sent Events - HTTP push mechanism for comparison.

MQTT Subscriptions - Topic-based pub/sub vs URI-based observe
WebSocket Server Push - Full-duplex vs observe’s asymmetric push
Server-Sent Events - HTTP push mechanism for comparison

Observe enables:

Read these points as one connected sequence: start with Event-Driven IoT - React to changes instead of polling; then Real-Time Monitoring - Dashboard updates without refresh; and finish with Smart Home Automation - Light sensors triggering actuators instantly.

Event-Driven IoT - React to changes instead of polling
Real-Time Monitoring - Dashboard updates without refresh
Smart Home Automation - Light sensors triggering actuators instantly

Implementation challenges:

Read these points as one connected sequence: start with NAT Traversal - UDP mapping timeouts requiring keepalives; then Sequence Number Management - Detecting out-of-order delivery; and finish with State Synchronization - Client-server observation lifecycle.

NAT Traversal - UDP mapping timeouts requiring keepalives
Sequence Number Management - Detecting out-of-order delivery
State Synchronization - Client-server observation lifecycle

Performance considerations:

Read these points as one connected sequence: start with Energy Optimization - NON vs CON for battery life (99% savings); then Bandwidth Management - Push vs poll bandwidth comparison; and finish with Scalability Patterns - Server memory per observer (50-100 bytes).

Energy Optimization - NON vs CON for battery life (99% savings)
Bandwidth Management - Push vs poll bandwidth comparison
Scalability Patterns - Server memory per observer (50-100 bytes)

14.8 See Also

CoAP Core Topics:

Read these points as one connected sequence: start with CoAP Fundamentals - Message types and reliability mechanisms; then CoAP Methods - GET with Observe: 0 option for registration; and finish with CoAP Security - Securing observe notifications with DTLS.

CoAP Fundamentals - Message types and reliability mechanisms
CoAP Methods - GET with Observe: 0 option for registration
CoAP Security - Securing observe notifications with DTLS

Implementation Guides:

Read these points as one connected sequence: start with CoAP Observe Implementation - Python and ESP32 code examples; then Observer Registry Design - Server-side state management; and finish with Rate Limiting Strategies - Preventing notification floods.

CoAP Observe Implementation - Python and ESP32 code examples
Observer Registry Design - Server-side state management
Rate Limiting Strategies - Preventing notification floods

Protocol Comparisons:

Read these points as one connected sequence: start with MQTT vs CoAP Observe - Topic subscriptions vs resource observation; then Long Polling vs Observe - HTTP alternative to push notifications; and finish with gRPC Streaming - Modern RPC with bidirectional streaming.

MQTT vs CoAP Observe - Topic subscriptions vs resource observation
Long Polling vs Observe - HTTP alternative to push notifications
gRPC Streaming - Modern RPC with bidirectional streaming

Real-World Applications:

Read these points as one connected sequence: start with Industrial IoT Monitoring - Vibration sensors with conditional observe; then Smart Agriculture - Soil moisture notifications on threshold changes; and finish with Building Automation - Temperature updates to thermostats.

Industrial IoT Monitoring - Vibration sensors with conditional observe
Smart Agriculture - Soil moisture notifications on threshold changes
Building Automation - Temperature updates to thermostats

Specifications & RFCs:

Read these points as one connected sequence: start with RFC 7641 - Observe Extension - Official specification; then RFC 7252 Section 5.10 - CoAP base protocol observe hooks; and finish with CoRE WG Documents - Latest IETF working group updates.

RFC 7641 - Observe Extension - Official specification
RFC 7252 Section 5.10 - CoAP base protocol observe hooks
CoRE WG Documents - Latest IETF working group updates

Debugging & Tools:

Read these points as one connected sequence: start with Wireshark CoAP Dissector Reference - Analyzing observe sequence numbers; then coap-client CLI Tool - Testing observe from command line; and finish with Californium Proxy - HTTP-CoAP observe bridging.

Wireshark CoAP Dissector Reference - Analyzing observe sequence numbers
coap-client CLI Tool - Testing observe from command line
Californium Proxy - HTTP-CoAP observe bridging

Match the Concepts

14.9 What’s Next

Now that you understand CoAP server push and the Observe extension, these chapters build directly on your knowledge:

CoAP Observe Registration and Freshness Contracts Focus: Token binding, Max-Age freshness, CON liveness checks, sequence ordering, and stale-notification handling Why read it: Turn Observe from a bandwidth-saving pattern into an auditable notification contract that recovers from loss, reordering, NAT expiry, and client restarts.
CoAP Advanced Features Focus: Block-wise transfer and large payload handling Why read it: Extend your Observe knowledge to handle firmware-update notifications that exceed a single UDP packet.
CoAP API Design Focus: RESTful URI patterns and resource modeling Why read it: Design well-structured observable resources that follow CoAP best practices for naming and content formats.
CoAP Security Applications Focus: Securing CoAP with DTLS and OSCORE Why read it: Protect observer registration and notification streams from eavesdropping and injection attacks.
CoAP Implementation Labs Focus: Python and ESP32 hands-on examples Why read it: Apply the registry and rate-limiting patterns from this chapter in working code on real hardware.
CoAP Message Types Focus: CON vs NON reliability and ACK/RST mechanics Why read it: Understand the reliability layer underpinning Observe notifications and how CON retransmission affects observer cleanup.
MQTT Broker and Topics Focus: Topic-based publish-subscribe with a broker Why read it: Compare Observe’s direct server-push model against MQTT’s broker-mediated fan-out for high-subscriber scenarios.

14.10 Summary

CoAP Observe turns a resource into a subscription-style stream while keeping RESTful resource semantics. Clients register interest, servers send ordered notifications, and both sides must handle sequence numbers, cancellation, and loss.

14.11 Continue Your Route

This final part closes the route from Interactive Calculator: NAT Keep-Alive Interval through Summary. Return to CoAP Observe: Subscriptions and Updates or continue from the coap module index.