10 Edge Acquisition: Missing Data and Field Design

10.1 Start With the Situation

A gateway translates and protects traffic correctly, but the field link still disappears and readings arrive late or not at all. The team must preserve the gap as evidence, buffer safely, and decide what may be aggregated before reconnection.

10.2 Overview

This route treats missing data explicitly, then applies field cases, store-and-forward design, aggregation, and radio-energy trade-offs.

This is part 2 of 2. Review Edge Acquisition: Power and Gateway Functions when you need the first route.

10.3 Learning Objectives

By the end of this chapter, you will be able to:

• represent missing data without silently deleting evidence
• design a bounded store-and-forward gateway
• compare aggregation and transmission energy in a field deployment

10.4 Chapter Roadmap

Follow the original sections below in order. They begin at the reviewed split boundary and keep every worked example, figure, check, and supporting banner with the section that owns it.

10.5 Handling Missing Data

# For slow-changing values (temperature): forward-fill
df['temp'] = df['temp'].ffill(limit=60)  # Max 60 missing samples

# For periodic values: interpolation
df['humidity'] = df['humidity'].interpolate(method='time')

# For event-driven sensors: mark as "no event" vs "missing"
df['motion_event'] = df['motion'].fillna(0)  # No motion detected
df['motion_missing'] = df['motion'].isna()    # Track sensor status

# For critical analysis: flag and include in model
df['has_missing'] = df.isna().any(axis=1)  # Create feature for missingness

10.6 Libelium Agriculture Gateway

The vineyard case shows why buffering, battery budgeting, and gateway placement have to be designed together.

10.7 Knowledge Check: Gateway Functions

10.8 Quiz 2: Gateway Functions

10.9 Visual Reference Gallery

Use these diagrams as alternate views of the same path: collect locally, translate or buffer, then transmit upstream.

10.10 AI-Generated Data Acquisition Diagrams

The chapter has treated acquisition, reduction, and gatewaying as separate design decisions. The four diagrams in Figure 10.1, Figure 10.2, Figure 10.3, and Figure 10.4 put those decisions beside one another so you can test whether they form one defensible path from a physical signal to an authenticated service request.

In Figure 10.1, read from the physical sensor through collection and local processing to transmission; the falling data rate is the evidence that reduction happens before the constrained link. Next inspect the deployment diagram in Figure 10.2 to locate the boundary that must implement that reduction without losing device meaning.

Trace Figure 10.2 from non-IP sensors through translation, buffering, and secure forwarding to cloud infrastructure. Those gateway functions explain how the first pipeline survives protocol differences and outages. The high-rate source behind that need becomes concrete in the time-series diagram in Figure 10.3.

Read Figure 10.3 across time and compare all accelerometer and gyroscope axes before deciding what a local window may summarise. The dense, related traces show why raw forwarding is expensive and why reduction must preserve decision-relevant motion. The final service boundary is shown in the API integration diagram in Figure 10.4.

Follow Figure 10.4 from edge-facing translation through authentication and formatting into the cloud service. The sequence shows that a syntactically valid conversion is not yet an admissible request: identity and schema checks still have to protect the service boundary. It also connects the field gateway to the application’s API contract, completing the end-to-end route from sampled signal to a request whose source, meaning, and permissions can be reviewed.

AI-generated SVG diagrams illustrating edge data acquisition concepts

Follow Figure 10.4 from edge-facing protocol translation through authentication and data formatting into the cloud service. It continues the gateway boundary into application integration rather than treating the upstream API as an unrelated endpoint. Together, the four diagrams connect the chapter’s running argument: a gateway is not merely a radio bridge; it preserves signal meaning while controlling volume, outages, protocol differences, and trust boundaries.

10.11 Practice Exercises

Store-and-Forward Gateway Exercise

Objective: Build a reliable gateway that buffers data during connectivity outages and implements intelligent synchronization.

The gateway photograph in Figure 10.5 identifies the hardware surfaces that must support this exercise’s buffer, local interfaces, and upstream synchronization. Use those surfaces to connect each task to a physical ingress, durable state, and egress path.

Photo: Laserlicht, CC BY-SA 4.0

Read Figure 10.5 from the local GPIO and peripheral interfaces to the processor, storage connection, and network ports. Sensor records enter on the field side, receive durable sequence and timestamp metadata in local storage, and leave through the network side after connectivity returns. The photograph does not itself prove reliability; it locates the resources the tasks must exercise. This ties the lab back to the running gateway contract: outage survival depends on software-defined buffering, ordering, retry, and acknowledgement evidence built on suitable hardware.

Tasks:

1. Set up edge gateway (Raspberry Pi) with local storage (SQLite database or time-series DB)
2. Implement data pipeline: sensors to gateway (buffer to DB) to cloud (MQTT or HTTP POST)
3. Add connectivity monitoring: ping cloud endpoint every 10s; if unreachable, set offline flag
4. Simulate 15-minute outage: disconnect network, accumulate 900 sensor samples (1 sample/sec); reconnect and verify all data syncs with correct timestamps

A small remote-data-logging version can start with a Raspberry Pi reading a DHT22 temperature and humidity sensor, stamping each accepted sample, saving it locally, and then sending the same record to a server or remote machine when the network path is available. Keep the payload boring and auditable: device id, sensor id, temperature, humidity, sample time, quality state, and upload attempt. That record lets the class separate three checks that are often confused in demos: the sensor was read, the gateway kept the reading, and the server received the same timestamped value.

Expected Outcome: Zero data loss during outage with proper temporal ordering after sync. Measure sync latency: 900 records should upload in <10 seconds. Understand buffer sizing: 1 GB storage handles ~10 million sensor readings at 100 bytes each = days of outages.

10.12 Interactive Quiz: Match Concepts

The remaining activities turn the design rules into recall and sequencing checks.

10.13 Interactive Quiz: Sequence the Steps

10.14 Label the Diagram

10.15 Code Challenge

10.16 Radio Budget and Aggregation

The radio module choice should be replayed as a full state sequence. Figure 10.6 prices wake, burst, sleep, and comparison under one 60-second soil-node workload.

In Figure 10.6, Measure the burst includes startup, listening, transmit, and ACK rather than quoting transmit current alone. Measure the floor shows why 6 µA versus 19 µA dominates when sleep occupies 59.75 s, and Compare equal work keeps payload, margin, retry policy, and reporting period fixed.

The follow-on page narrows the lens to radio scheduling and aggregation contracts.

For the deeper implementation contract behind radio duty cycle, adaptive sampling, gateway aggregation, batching latency, event-preserving summaries, and queue metadata, continue to Radio Budget and Gateway Aggregation Contracts.

10.17 Summary

Power management and gateway functions are critical for practical edge IoT deployments:

• Duty cycling: Maximize sleep time and batch transmissions - TX current dominates power budget
• Battery life: Optimized duty cycling extends battery life from months to years (4-5x improvement typical)
• Gateway functions: Protocol translation, store-and-forward buffering, and security are essential for non-IP device integration
• Missing data: Treat as information, not noise - use appropriate imputation and track as a health metric

10.18 Concept Relationships

Power management and gateways are critical constraints that determine edge system feasibility:

Power Optimization (This chapter):

• Duty cycling formula extends battery life from months to years (4-5x improvement via deep sleep)
• Transmission dominates power budget (10-100x more than sensing); batching is highest-leverage optimization

Gateway Functions (This chapter):

• Protocol translation bridges Non-IP devices to cloud (Modbus → MQTT, Zigbee → HTTP)
• Store-and-forward prevents data loss during network outages (24-48 hour buffer minimum)

Architecture Foundation:

• Edge Data Acquisition: Architecture - Non-IP Things require gateways; power budgets vary by device category
• Edge Data Acquisition: Sampling and Compression - Compression reduces transmission frequency, directly impacting battery life

Broader Context:

• Edge Compute Patterns - Edge processing reduces transmission (saves power); offloading to fog/cloud (costs power)
• Edge Fog Computing - Gateways often serve as fog nodes, providing both connectivity and local processing

Data Quality Impact:

• Data Quality and Preprocessing - Gateway store-and-forward ensures zero data loss (vs gaps during outages)
• Multi-Sensor Data Fusion - Gateways often perform sensor fusion locally, reducing bandwidth

Key Insight: Power and connectivity are linked — aggressive transmission schedules drain batteries fast. Vineyard case study: Gateways with store-and-forward achieved 0.02% data loss (vs 8% without), while duty cycling achieved 5.4-year battery life (vs target of 3 years). The combination makes low-power, high-reliability IoT feasible.

10.19 What’s Next

10.20 See Also

Edge Acquisition Series:

• Edge Data Acquisition: Architecture - Device categories and connectivity paths
• Edge Data Acquisition: Sampling and Compression - Data reduction techniques

Edge Computing:

• Edge Compute Patterns - Processing patterns and optimization
• Edge Fog Computing - Three-tier architecture

Data Quality:

• Data Quality and Preprocessing - Validation and cleaning pipelines
• Multi-Sensor Data Fusion - Combining data from multiple sources

Practice:

• Edge Topic Review - Edge computing review
• Edge Comprehensive Review - Complete concepts review
• Edge Quiz Bank - Knowledge testing

10.21 Additional Resources

• AWS IoT Greengrass Documentation - Edge runtime for local compute and data caching
• Azure IoT Edge - Cloud intelligence deployed locally
• ESP-IDF Power Management - ESP32 power optimization