6 Data Encoding and Codecs

6.1 Start With the Data Nobody Can Read

Store a Sample That Tomorrow’s Tool Can Explain

Picture a cold-room camera and temperature unit after a fault. The files exist, but the review tool cannot match the clip to the reading or tell which unit the number used.

Telemetry means facts a device sends so its state can be observed. Keep a small known sample with its source, time, unit, quality, format name, and version. Decode it with a second tool before release.

Latency means the time from an event to the result that needs it. Measure it from capture to the visible chart or image. Compare size, lost detail, decode work, and delay with written pass limits.

Remove one field, use an old reader, shift the time, and damage part of the file. The screen should show the gap or doubt. It should not quietly draw a confident story from unknown bytes.

One sample cannot cover every device or tool. Practitioner compares encodings and media choices. Under the Hood follows compression, timing, and decode limits in detail.

A camera feed, a microphone clip, and a temperature stream can all reach storage successfully and still be useless if the next system cannot decode them with the right time, units, and metadata. Encoding choices matter because they decide whether a dashboard, replay tool, or investigation can turn stored bytes back into trustworthy evidence.

6.2 In 60 Seconds

Visualization depends on data arriving in a form that can be decoded, trusted, aligned in time, and displayed. Encoding is the way data is represented for transport or storage. A codec is a method for encoding and decoding signals such as audio, image, or video. A container is a file or stream wrapper that can hold encoded media and metadata.

For IoT visualization, encoding choices affect:

• dashboard latency
• network use
• edge compute load
• browser or tool compatibility
• data fidelity
• timestamp accuracy
• storage and replay
• whether the visualization can explain uncertainty or stale state

The right choice starts with the visualization question. A fault dashboard, a wildlife microphone, a security camera, and a long-term analytics archive need different encodings.

6.3 Learning Objectives

By the end of this chapter, you will be able to:

• distinguish data encoding, codec, and container
• choose between human-readable and compact telemetry encodings
• explain lossy and lossless encoding tradeoffs for IoT visualization
• identify when media codecs matter for camera, microphone, and image streams
• preserve metadata needed for trustworthy visualizations
• evaluate encoding choices by fidelity, latency, compute, compatibility, and replay needs

6.4 Quick Check: Data Encoding and Codecs

6.5 Encoding, Codec, And Container

These terms are related but not interchangeable.

Encoding: how data is represented. Examples include JSON, CSV, CBOR, Protocol Buffers, JPEG, PNG, Opus, H.264, and H.265. Codec: a coder-decoder used for signals such as audio, image, or video. It compresses data for transport or storage and decodes it for playback or analysis. Container: a wrapper that holds encoded streams and metadata. Examples include MP4, WebM, WAV, and Matroska.

Common confusion:

MP4 is a container, not a video codec. H.264 is a video codec, not a container. JSON is a telemetry encoding, not a media codec. A dashboard may use JSON for sensor values and H.264 for a camera view at the same time.

6.6 Visualization Encoding Pipeline

Before comparing formats, follow one observation through every representation change. Figure 6.1 shows where meaning can be added, lost, or misread between sensing and the final display.

In Figure 6.1, start with the sensed value and its unit, then inspect the encoder’s schema and the transport envelope. Storage must retain enough metadata for the decoder to reconstruct that meaning, and the visual layer must expose freshness and quality rather than only a number. This end-to-end reading is the basis for every codec trade-off that follows.

6.7 Telemetry Encodings

Most IoT dashboards begin with telemetry: timestamped records such as temperature, pressure, location, status, battery state, or event counts.

Common options:

JSON: easy to inspect and widely supported. Good for prototypes, APIs, and debugging. Larger than binary encodings. CSV: compact for flat rows and easy to import. Weak for nested state, units, schemas, and event meaning. CBOR: compact binary form that maps well to JSON-like structures. Useful when devices need smaller payloads but flexible fields. Protocol Buffers: compact schema-based records. Useful when producer and consumer can share a stable schema. Apache Arrow or Parquet: useful for analytics and column-oriented storage after ingestion, not usually a constrained-device payload format.

Choose based on the visualization task:

Human debugging favors readable formats. Battery and bandwidth constrained links favor compact formats. Long-term analytics favors schema and column structure. Mixed-device dashboards need clear versioning and metadata.

6.8 Payload Budget Example

Small payloads change battery life, airtime, and retry behavior. Suppose protocol and security overhead leave 51 bytes of application payload for each uplink. A compact JSON representation of one small reading may cost about 32 bytes, so it fits one reading and leaves little room for timestamp, unit, or quality fields. A CBOR representation at about 18 bytes can fit two comparable readings in the same payload. A raw binary record at about 4 bytes can fit many readings, but only if every receiver already knows the field order, scale factor, byte order, status mapping, and schema version. Read the comparison in Figure 6.2 from the most inspectable encoding toward the most compact.

Reading Figure 6.2 from JSON through CBOR to raw binary shows that JSON consumes the most bytes but leaves field meaning visible, CBOR preserves a structured model with less airtime, and raw binary packs the most readings into the budget by moving meaning into an external contract. That progression explains why the smallest payload is not automatically best: the design must pay for schema control, decoder compatibility, diagnostics, and rollback somewhere.

6.9 Metadata Is Part Of The Encoding

A value without context can mislead the visualization.

Include:

• timestamp and clock source
• unit
• device identity
• sensor type
• location or asset
• sampling interval
• quality flag
• current, stale, estimated, or missing state
• calibration or firmware version when relevant
• coordinate reference for maps

Do not rely on the chart title to carry this information. The data record should preserve enough meaning for the chart, alert, replay, and support path.

6.10 Lossless And Lossy Choices

Lossless encoding preserves the original data exactly after decoding. Lossy encoding discards some information to reduce size or bitrate.

Use lossless or near-lossless approaches when:

records are used for audit or investigation downstream analysis needs exact values small changes matter calibration or diagnosis is required the signal will be transformed more than once

Use lossy approaches when:

humans are visually inspecting images or video bandwidth is limited approximate visual quality is acceptable the stream must be viewed in real time retention length matters more than exact reconstruction

For visualization, the key question is not “Which codec is best?” The key question is “What information must survive decoding for the task?”

6.11 Numeric Time-Series Compression

Time-series storage often gets large savings without losing exact raw values. The trick is exploiting regularity column by column. Timestamps are near-periodic, so delta-of-delta encoding stores the change in the interval rather than every full timestamp. A steady 10-second cadence becomes a long run of zeros after the first values. Numeric values can use Gorilla-style XOR compression: each floating-point value is compared with its predecessor, and because adjacent sensor readings often differ only slightly, the XOR has long runs of leading and trailing zero bits.

This only works well when the storage layout groups similar values together. A column of timestamps, a column of temperatures, and a column of quality flags each give the encoder a predictable pattern. A row that mixes timestamps, JSON blobs, identifiers, units, and arbitrary strings gives the codec much less to exploit.

For a 10-second sensor, timestamp deltas are usually 10, 10, 10, 10. Delta-of-delta stores the change between those deltas: 0, 0, 0 after the first value. If one packet arrives late and the interval becomes 12 seconds, the encoder records a small exception and then returns to zeros when the cadence stabilizes. That is different from storing every full timestamp, and it is still lossless.

Separate three decisions in the storage plan:

• use lossless column compression for the raw audit window so incidents can be replayed exactly;
• use rollups or downsampling only after the team defines which questions no longer need raw samples;
• keep schema and unit metadata beside compressed values so a dashboard can distinguish a physical change from an encoding, unit, or calibration change.

6.12 Media Codecs For IoT Visualization

Media streams include camera video, still images, microphone audio, vibration audio, and spectrogram-like outputs.

Before choosing a codec, classify the media workload. Text, still images, and graphics are discrete media; audio, video, and animation are continuous media. A stream may also be captured or synthesized, local or networked, and independently manipulable even when several media types are presented together. Those attributes change more than storage size: they affect representation, indexing and search, real-time scheduling, authoring, user interaction, protection, distribution, and session control. Record which of those responsibilities the IoT product actually needs instead of treating “multimedia” as one undifferentiated payload type.

The same classification explains why bandwidth alone did not create networked multimedia. The source connects wider wired and wireless access with faster processors and storage, new compression and vision algorithms, increasingly capable terminals, and the digitization of capture and playback devices. In an IoT review, test the whole chain: capture, processing, storage, network, presentation, and interaction can each become the limiting stage.

A raw-media budget check explains why codecs show up so quickly in IoT systems. A 16 kHz, 14-bit mono audio stream is already about 224 kbps before headers. CD-quality stereo audio is about 1.4 Mbps. A single 1024 by 1024 RGB image is roughly 3 MB before compression, and 720p RGB video at 24 frames per second is above 500 Mbps. Those numbers are not release evidence by themselves, but they force the review to name where compression happens and what information may be lost.

A useful compression review separates the stages. Time-frequency or spatial decomposition exposes structure such as DCT, wavelet, short-time Fourier, or similar transform coefficients. Quantization is the lossy step that spends fewer bits on less important detail. Entropy coding, such as Huffman-style or arithmetic coding, is the lossless packing step that assigns shorter codes to frequent symbols. For audio this may depend on psychoacoustic masking; for images, JPEG-style paths commonly convert RGB display values into luma/chroma channels before discarding or subsampling detail. The deployment record should say whether the lost detail matters for inspection, alarm review, or later audit.

Common image and video choices:

• JPEG: useful for still images where small file size matters and minor visual loss is acceptable.
• PNG: useful for diagrams, masks, labels, or evidence images where exact edges matter.
• MJPEG: simple stream of JPEG frames. Easy to decode but inefficient for continuous video.
• H.264: widely supported video codec for live viewing and browser compatibility.
• H.265: often more compact than H.264 but may require more compute and may have compatibility constraints.
• AV1: efficient modern codec, often better suited to capable edge or cloud processing than small devices.

For video, the review should also name the prediction structure rather than only the codec brand. An I-frame is self-contained enough to become a seek point or recovery point. P-frames predict from earlier pictures, and B-frames can use past and future references, which improves compression but can add buffering and delay. Motion estimation, block size, transform choice, quantization, run-length coding, and entropy coding all affect the same deployment question: whether the retained stream still supports the inspection, control, or audit task. If the product needs low-latency viewing, short recovery after packet loss, or frame-accurate evidence, the GOP length and reference-frame policy belong in the release record.

Common audio choices:

• PCM: uncompressed audio; useful for reference capture and diagnosis but large.
• FLAC: lossless compressed audio for archival or analysis.
• Opus: flexible speech and audio codec with good low-latency behavior.
• AAC: common compressed audio for media playback.

The visualization should state when users are seeing compressed media, sampled media, delayed media, or representative thumbnails.

Quality evaluation should match the media claim. PSNR and SSIM can compare images or video against a reference, but they do not prove that a human operator sees the difference that matters. Audio reviews may use subjective tests such as ABX, ABC/HR, or MUSHRA when listeners must judge audible differences, and objective models such as PEAQ when a perceptual estimate is acceptable for screening. Speech-codec work may instead name PESQ (ITU-T P.862) or POLQA (ITU-T P.863); the source shows why the metric must travel with the codec and operating condition by comparing a PESQ result with mean-opinion scores under zero frame erasure. The release note should keep the test method, reference material, codec settings, network impairment, and pass/fail threshold together; otherwise a single quality score can hide the artifact that breaks the application.

Older multimedia vocabulary is still useful when reviewing legacy demos, SDKs, or vendor claims. Treat the names as evidence prompts rather than as automatic design choices:

• Computer vision: image analysis that should record frame source, model or rule version, latency, and failure examples. A camera demo is not a vision system until the team can name the capture path, preprocessing step, feature or model decision, and the action that follows. Frame cameras and event-style cameras such as dynamic vision sensors create different data streams, so the review should preserve the timing, threshold, and calibration evidence that makes the interpretation reproducible. Libraries such as OpenCV, SimpleCV, or vendor SDKs can shorten implementation, but they do not replace evidence about lighting, motion blur, occlusion, frame rate, false positives, and whether the result is good enough for the IoT decision.
• Camera lab setup: prototype evidence should say which image library stack was installed, which camera device or stream was opened, and how the capture path was tested. A Linux board exercise might use Python imaging libraries, OpenCV or SimpleCV wrappers, and a /dev/video* camera, but the release record still needs a repeatable camera check, dependency list, failure notes for missing imaging libraries, and any server or scheduled-capture process that makes pictures available remotely.
• Native SDKs: platform toolkits, such as an Android NDK or board vendor SDK, that need pinned versions, build flags, and device compatibility evidence.
• Multimedia and home-platform stacks: terms such as multimedia home platform, MIDI, real media, video telephony, DirectX, OpenGL, JavaScript, and QuickTime point to different layers: content format, application API, rendering API, scripting layer, or container/player behavior. The review should name which layer is responsible for decoding, rendering, user interaction, and fallback.
• Interframe coding and delta frames: video compression that stores changes between frames, so loss, seeking, and key-frame interval evidence matter.
• Subsampling and color spaces: reductions such as chroma subsampling or sRGB conversion can save bytes, but the release record should prove that labels, safety colors, and inspection details still survive.
• Display artifacts: tearing, overlay composition, motion-video smoothness, and video mapping are rendering-path issues, not sensor facts. Confirm refresh timing, GPU or browser support, and what happens when the display cannot keep up.

6.13 Codec Decision Map

Codec selection starts with the evidence the application must preserve. Use Figure 6.3 to move through fidelity and latency first, then test the surviving choices against bandwidth, compute, compatibility, metadata, and replay needs.

In Figure 6.3, fidelity rejects encodings that discard evidence the task needs, while latency removes choices whose processing or buffering misses the response window. Bandwidth and device compute then constrain what can be transported and decoded. Compatibility, metadata, and replay complete the route by asking whether future readers can interpret and investigate the record. The result is a defendable contract, not a codec popularity vote.

6.14 Latency And Real-Time Display

Real-time visualization is not only about fast networks. Encoding can add delay.

Check:

• capture delay
• encoding delay
• transport buffering
• broker or server buffering
• decoding delay
• dashboard refresh behavior
• browser playback support

For a live control view, choose formats and settings that preserve timely feedback. For an archive view, higher compression and slower encoding may be acceptable.

6.15 Compatibility And Decode Paths

An encoding choice fails if the receiver cannot decode it reliably.

Check:

• device encoder support
• gateway transcoding needs
• browser playback support
• dashboard library support
• analytics tool support
• mobile device support
• hardware acceleration availability
• fallback format for older clients

Avoid making every viewer depend on a rare codec unless the deployment controls the full device and software stack.

On Android-style endpoints, make the media path explicit in the same way. A MediaPlayer-class wrapper may accept a local file, URI, or URL and hide the decoder details, but the review still needs to record supported formats, buffering behavior, network permissions, error callbacks, and what happens when storage or connectivity disappears. An AudioManager-style service owns audio focus and output routing, so a kiosk, wearable, or gateway display should prove whether alerts, calls, media streams, Bluetooth devices, and other apps can interrupt or duck each other safely. Treat those classes as responsibility boundaries, not as proof that playback will be reliable in the final device context.

6.16 Worked Example: Environmental Dashboard

Scenario:

• A building dashboard shows temperature, humidity, occupancy, and device health.
• Operators need current state, trend, and stale-data awareness.

Encoding approach:

• Use schema-backed telemetry records for device readings.
• Include timestamp, unit, room, device identity, and quality state.
• Keep current dashboard payloads small.
• Store historical records in an analytics-friendly format after ingestion.
• Make stale and missing state explicit rather than treating missing values as zero.

Visualization result:

• The dashboard can show current values, trend history, stale readings, and device health without guessing what each field means.

6.17 Worked Example: Remote Camera View

Scenario:

• A remote camera shows periodic images and occasional live video.
• Users need enough detail to inspect events, but the link is constrained.

Encoding approach:

• Use still images or low frame rate previews for routine monitoring.
• Use a widely supported video codec for live inspection.
• Preserve event time, camera identity, location, and quality state with the media.
• Store key frames or clips for replay if the event needs review.
• Show users whether the view is live, delayed, or a saved clip.

Visualization result:

• Users can distinguish a current stream from replay, and the system avoids sending more media than the task requires.

6.18 Review Checklist

Before accepting a visualization encoding design, confirm:

the visualization question is clear the encoding can carry timestamp, unit, identity, and quality state the decoder path is known for every viewer lossy choices preserve the information needed for the task real-time views account for encoding and buffering delay archive formats support replay and analysis stale, missing, estimated, and current states remain distinguishable fallback behavior is defined when a client cannot decode a stream

6.19 Common Defects

Watch for:

Container-codec confusion: assuming MP4 support means every video stream inside it will play. Missing metadata: plotting values without units, device identity, or freshness. Over-readable payloads: using large text payloads on constrained links only because they are easy to debug. Over-compact payloads: using binary records without schema, versioning, or inspection tools. Lossy analysis damage: compressing signals in a way that removes features needed by analytics. Hidden delay: presenting delayed media as if it were live. No fallback: choosing a codec that some dashboards or browsers cannot decode.

6.20 Knowledge Check

6.21 Matching Quiz

6.22 Ordering Quiz

6.23 Summary

Data encoding and codecs shape what an IoT visualization can show, how quickly it can show it, and how much users can trust it. Telemetry records need schema, timestamp, unit, identity, and quality state. Media streams need codec and container choices that match fidelity, latency, compute, compatibility, and replay needs.

6.24 Key Takeaway

Visualization codecs should preserve time, units, quality, and structure so dashboards do not misrepresent telemetry.

6.25 Concept Relationships

This chapter connects to:

Visualization Overview for why data must be understandable quickly. Visualization Types for matching data shape to chart type. Dashboard Design for presenting encoded data safely. Real-Time Visualization for latency and streaming behavior. Visualization Tools for tool compatibility.

6.26 What’s Next

Continue with:

Visualization Tools, if you need to choose dashboard or stream tooling. Real-Time Visualization, if encoding delay affects the view. Dashboard Design, if users need to interpret encoded telemetry safely.