10 Edge Acquisition: Missing Data and Field Design

10.1 Start With the Situation

This section: Start With the Situation — A gateway passes data between device and wider network.

A gateway passes data between device and wider network. The field link is the connection from the remote site. Even if the gateway translates and protects traffic, that link can disappear and readings can arrive late or not at all. The team must keep the gap as evidence. A buffer holds readings until reconnection; aggregation combines readings into a smaller report when the task allows it.

10.2 Overview

This section: Overview — This route treats missing data explicitly, then applies field cases, store-and-forward design, aggregation, and radio-energy trade-offs.

This route treats missing data explicitly, then applies field cases, store-and-forward design, aggregation, and radio-energy trade-offs.

This is part 2 of 2. Review Edge Acquisition: Power and Gateway Functions when you need the first route.

10.3 Learning Objectives

This section: Learning Objectives

By the end of this chapter, you will be able to:

• represent missing data without silently deleting evidence
• design a bounded store-and-forward gateway
• compare aggregation and transmission energy in a field deployment

10.4 Handling Missing Data

This section: Handling Missing Data

# For slow-changing values (temperature): forward-fill
df['temp'] = df['temp'].ffill(limit=60)  # Max 60 missing samples

# For periodic values: interpolation
df['humidity'] = df['humidity'].interpolate(method='time')

# For event-driven sensors: mark as "no event" vs "missing"
df['motion_event'] = df['motion'].fillna(0)  # No motion detected
df['motion_missing'] = df['motion'].isna()    # Track sensor status

# For critical analysis: flag and include in model
df['has_missing'] = df.isna().any(axis=1)  # Create feature for missingness

10.5 Illustrative Vineyard Gateway

This section: Illustrative Vineyard Gateway — The vineyard case shows why buffering, battery budgeting, and gateway placement have to be designed together.

The vineyard case shows why buffering, battery budgeting, and gateway placement have to be designed together.

10.6 Knowledge Check: Gateway Functions

This section: Knowledge Check: Gateway Functions

10.7 Quiz 2: Gateway Functions

This section: Quiz 2: Gateway Functions

10.8 Visual Reference Gallery

This section: Visual Reference Gallery — Use these diagrams as alternate views of the same path: collect locally, translate or buffer, then transmit upstream.

Use these diagrams as alternate views of the same path: collect locally, translate or buffer, then transmit upstream.

10.9 AI-Generated Data Acquisition Diagrams

This section: AI-Generated Data Acquisition Diagrams — The chapter has treated acquisition, reduction, and gatewaying as separate design decisions.

The chapter has treated acquisition, reduction, and gatewaying as separate design decisions. The four diagrams in Figure 10.1, Figure 10.2, Figure 10.3, and Figure 10.4 put those decisions beside one another so you can test whether they form one defensible path from a physical signal to an authenticated service request.

+

After locating data reduction in Figure 10.1, inspect Figure 10.2 to ask which gateway boundary must shrink the stream without losing device meaning.

+

Begin at the non-IP sensors in Figure 10.2 and ask what must survive the gateway boundary. Translation handles protocol differences, but it must keep each reading’s device meaning intact. The buffer then carries that meaning through a field-link outage instead of turning missing connectivity into missing history. Secure forwarding resumes only when the upstream path returns. Read the outage strip last: late readings and an observed gap are both evidence, so replay must not make the stream look continuous when it was not. The inertial-stream view that follows shows how quickly this duty grows for a high-rate source.

+

Read Figure 10.3 across time and compare all accelerometer and gyroscope axes before deciding what a local window may summarise. The dense, related traces show why raw forwarding is expensive and why reduction must preserve decision-relevant motion. The final service boundary is shown in the API integration diagram in Figure 10.4.

+

Follow Figure 10.4 from edge-facing translation through authentication and formatting into the cloud service. The sequence shows that a syntactically valid conversion is not yet an admissible request: identity and schema checks still have to protect the service boundary. It also connects the field gateway to the application’s API contract, completing the end-to-end route from sampled signal to a request whose source, meaning, and permissions can be reviewed.

AI-generated SVG diagrams illustrating edge data acquisition concepts

Follow Figure 10.4 from edge-facing protocol translation through authentication and data formatting into the cloud service. It continues the gateway boundary into application integration rather than treating the upstream API as an unrelated endpoint. Together, the four diagrams connect the chapter’s running argument: a gateway is not merely a radio bridge; it preserves signal meaning while controlling volume, outages, protocol differences, and trust boundaries.

10.10 Practice Exercises

This section: Practice Exercises

Store-and-Forward Gateway Exercise

Objective: Build a reliable gateway that buffers data during connectivity outages and implements intelligent synchronization.

The gateway photograph in Figure 10.5 identifies the hardware surfaces that must support this exercise’s buffer, local interfaces, and upstream synchronization. Use those surfaces to connect each task to a physical ingress, durable state, and egress path.

+

Photo: Laserlicht, CC BY-SA 4.0

Read Figure 10.5 from the local GPIO and peripheral interfaces to the processor, storage connection, and network ports. Sensor records enter on the field side, receive durable sequence and timestamp metadata in local storage, and leave through the network side after connectivity returns. The photograph does not itself prove reliability; it locates the resources the tasks must exercise. This ties the lab back to the running gateway contract: outage survival depends on software-defined buffering, ordering, retry, and acknowledgement evidence built on suitable hardware.

Tasks:

1. Set up edge gateway (Raspberry Pi) with local storage (SQLite database or time-series DB)
2. Implement data pipeline: sensors to gateway (buffer to DB) to cloud (MQTT or HTTP POST)
3. Add connectivity monitoring: ping cloud endpoint every 10s; if unreachable, set offline flag
4. Simulate 15-minute outage: disconnect network, accumulate 900 sensor samples (1 sample/sec); reconnect and verify all data syncs with correct timestamps

A small remote-data-logging version can start with a Raspberry Pi reading a DHT22 temperature and humidity sensor, stamping each accepted sample, saving it locally, and then sending the same record to a server or remote machine when the network path is available. Keep the payload boring and auditable: device id, sensor id, temperature, humidity, sample time, quality state, and upload attempt. That record lets the class separate three checks that are often confused in demos: the sensor was read, the gateway kept the reading, and the server received the same timestamped value.

Expected Outcome: Zero data loss during outage with proper temporal ordering after sync. Measure sync latency: 900 records should upload in <10 seconds. Understand buffer sizing: 1 GB storage handles ~10 million sensor readings at 100 bytes each = days of outages.

10.11 Interactive Quiz: Match Concepts

This section: Interactive Quiz: Match Concepts — The remaining activities turn the design rules into recall and sequencing checks.

The remaining activities turn the design rules into recall and sequencing checks.

10.12 Interactive Quiz: Sequence the Steps

This section: Interactive Quiz: Sequence the Steps

10.13 Label the Diagram

This section: Label the Diagram

10.14 Code Challenge

This section: Code Challenge

10.15 Radio Budget and Aggregation

This section: Radio Budget and Aggregation — The radio module choice should be replayed as a full state sequence.

The radio module choice should be replayed as a full state sequence. Figure 10.6 prices wake, burst, sleep, and comparison under one 60-second soil-node workload.

+

In Figure 10.6, Measure the burst includes startup, listening, transmit, and ACK rather than quoting transmit current alone. Measure the floor shows why 6 µA versus 19 µA dominates when sleep occupies 59.75 s, and Compare equal work keeps payload, margin, retry policy, and reporting period fixed.

The follow-on page narrows the lens to radio scheduling and aggregation contracts.

For the deeper implementation contract behind radio duty cycle, adaptive sampling, gateway aggregation, batching latency, event-preserving summaries, and queue metadata, continue to Radio Budget and Gateway Aggregation Contracts.

10.16 Summary

This section: Summary

Power management and gateway functions are critical for practical edge IoT deployments:

• Duty cycling: Maximize sleep time and batch transmissions - TX current dominates power budget
• Battery life: In this chapter’s duty-cycle example, optimization extends calculated battery life from months to years (about 4.9x).
• Gateway functions: Protocol translation, store-and-forward buffering, and security are essential for non-IP device integration
• Missing data: Treat as information, not noise - use appropriate imputation and track as a health metric

10.17 Concept Relationships

This section: Concept Relationships — Key Insight: Power and connectivity are linked — aggressive transmission schedules drain batteries fast.

Power management and gateways are critical constraints that determine edge system feasibility:

Power Optimization (This chapter):

• In this chapter’s duty-cycle example, deep sleep extends calculated battery life from months to years (about 4.9x).
• Transmission can dominate the power budget; batching is often a useful optimization when wake-up and handshake costs matter.

Gateway Functions (This chapter):

• Protocol translation bridges Non-IP devices to cloud (Modbus → MQTT, Zigbee → HTTP)
• Store-and-forward prevents data loss during network outages (24-48 hour buffer minimum)

Architecture Foundation:

• Edge Data Acquisition: Architecture - Non-IP Things require gateways; power budgets vary by device category
• Edge Data Acquisition: Sampling and Compression - Compression reduces transmission frequency, directly impacting battery life

Broader Context:

• Edge Compute Patterns - Edge processing reduces transmission (saves power); offloading to fog/cloud (costs power)
• Edge Fog Computing - Gateways often serve as fog nodes, providing both connectivity and local processing

Data Quality Impact:

• Data Quality and Preprocessing - Gateway store-and-forward ensures zero data loss (vs gaps during outages)
• Multi-Sensor Data Fusion - Gateways often perform sensor fusion locally, reducing bandwidth

Key Insight: Power and connectivity are linked — aggressive transmission schedules drain batteries fast. In the illustrative vineyard scenario, gateways with store-and-forward reduce assumed data loss from 8% to 0.02%, while the duty-cycle calculation gives 5.4 years of battery life against a three-year target. The combination makes low-power, high-reliability IoT feasible.

10.18 What’s Next

This section: What’s Next

10.19 See Also

This section: See Also

Edge Acquisition Series:

• Edge Data Acquisition: Architecture - Device categories and connectivity paths
• Edge Data Acquisition: Sampling and Compression - Data reduction techniques

Edge Computing:

• Edge Compute Patterns - Processing patterns and optimization
• Edge Fog Computing - Three-tier architecture

Data Quality:

• Data Quality and Preprocessing - Validation and cleaning pipelines
• Multi-Sensor Data Fusion - Combining data from multiple sources

Practice:

• Edge Topic Review - Edge computing review
• Edge Comprehensive Review - Complete concepts review
• Edge Quiz Bank - Knowledge testing

10.20 Additional Resources

This section: Additional Resources

• AWS IoT Greengrass Documentation - Edge runtime for local compute and data caching
• Azure IoT Edge - Cloud intelligence deployed locally
• ESP-IDF Power Management - ESP32 power optimization