Data Storage · Study deck
Data Encoding and Codecs
Picture a cold-room camera and temperature unit after a fault.
Data Dora is your guide for this deck.
After studying this chapter
Learning objectives
You will be able to:
- Explain: A camera feed, a microphone clip, and a temperature stream can all reach storage successfully and still be useless if the next system cannot decode them with the right time, units, and metadata.
- Explain: Motion estimation, block size, transform choice, quantization, run-length coding, and entropy coding all affect the same deployment question: whether the retained stream still supports the inspection, control, or audit task.
- Explain: humans are visually inspecting images or video bandwidth is limited approximate visual quality is acceptable the stream must be viewed in real time retention length matters more than exact reconstruction.
Major section
Start With the Data Nobody Can Read
The screen should show the gap or doubt.
- The files exist, but the review tool cannot match the clip to the reading or tell which unit the number used.
- Telemetry means facts a device sends so its state can be observed.: Keep a small known sample with its source, time, unit, quality, format name, and version.
- It should not quietly draw a confident story from unknown bytes.
- One sample cannot cover every device or tool.
Major section
Start With the Data Nobody Can Read (continued)
Latency means the time from an event to the result that needs it.: Measure it from capture to the visible chart or image.
- Under the Hood follows compression, timing, and decode limits in detail.
- A camera feed, a microphone clip, and a temperature stream can all reach storage successfully and still be useless if the next system cannot decode them with the right time, units, and metadata.
- Encoding choices matter because they decide whether a dashboard, replay tool, or investigation can turn stored bytes back into trustworthy evidence.
Major section
In 60 Seconds
Visualization depends on data arriving in a form that can be decoded, trusted, aligned in time, and displayed.
- Encoding is the way data is represented for transport or storage.
- A container is a file or stream wrapper that can hold encoded media and metadata.
- The right choice starts with the visualization question.
Major section
Encoding, Codec, And Container
A dashboard may use JSON for sensor values and H.264 for a camera view at the same time.
- These terms are related but not interchangeable.
- Encoding: how data is represented.
- Container: a wrapper that holds encoded streams and metadata.
- MP4 is a container, not a video codec.
Major section
Visualization Encoding Pipeline
Storage must retain enough metadata for the decoder to reconstruct that meaning, and the visual layer must expose freshness and quality rather than only a number.
- This end-to-end reading is the basis for every codec trade-off that follows.
Major section
Payload Budget Example
Small payloads change battery life, airtime, and retry behavior.
- A compact JSON representation of one small reading may cost about 32 bytes, so it fits one reading and leaves little room for timestamp, unit, or quality fields.
- A CBOR representation at about 18 bytes can fit two comparable readings in the same payload.
Major section
Lossless And Lossy Choices
Lossy encoding discards some information to reduce size or bitrate.
- Lossless encoding preserves the original data exactly after decoding.
- records are used for audit or investigation downstream analysis needs exact values small changes matter calibration or diagnosis is required the signal will be transformed more than once.
- humans are visually inspecting images or video bandwidth is limited approximate visual quality is acceptable the stream must be viewed in real time retention length matters more than exact reconstruction.
Major section
Numeric Time-Series Compression
The trick is exploiting regularity column by column.
- Timestamps are near-periodic, so delta-of-delta encoding stores the change in the interval rather than every full timestamp.
- A steady 10-second cadence becomes a long run of zeros after the first values.
- This only works well when the storage layout groups similar values together.
Major section
Media Codecs For IoT Visualization
A raw-media budget check explains why codecs show up so quickly in IoT systems.
- A stream may also be captured or synthesized, local or networked, and independently manipulable even when several media types are presented together.
- The same classification explains why bandwidth alone did not create networked multimedia.
- A useful compression review separates the stages.
Major section
Media Codecs For IoT Visualization (continued)
The source connects wider wired and wireless access with faster processors and storage, new compression and vision algorithms, increasingly capable terminals, and the digitization of capture and playback devices.
- In an IoT review, test the whole chain: capture, processing, storage, network, presentation, and interaction can each become the limiting stage.
- A 16 kHz, 14-bit mono audio stream is already about 224 kbps before headers.
- CD-quality stereo audio is about 1.4 Mbps.
Major section
Media Codecs For IoT Visualization (continued)
A single 1024 by 1024 RGB image is roughly 3 MB before compression, and 720p RGB video at 24 frames per second is above 500 Mbps.
- Those numbers are not release evidence by themselves, but they force the review to name where compression happens and what information may be lost.
- Quantization is the lossy step that spends fewer bits on less important detail.
- Quality evaluation should match the media claim.
Major section
Media Codecs For IoT Visualization (continued)
Entropy coding, such as Huffman-style or arithmetic coding, is the lossless packing step that assigns shorter codes to frequent symbols.
- For audio this may depend on psychoacoustic masking; for images, JPEG-style paths commonly convert RGB display values into luma/chroma channels before discarding or subsampling detail.
- The deployment record should say whether the lost detail matters for inspection, alarm review, or later audit.
- For video, the review should also name the prediction structure rather than only the codec brand.
- An I-frame is self-contained enough to become a seek point or recovery point.
Major section
Media Codecs For IoT Visualization (continued)
P-frames predict from earlier pictures, and B-frames can use past and future references, which improves compression but can add buffering and delay.
- Motion estimation, block size, transform choice, quantization, run-length coding, and entropy coding all affect the same deployment question: whether the retained stream still supports the inspection, control, or audit task.
- If the product needs low-latency viewing, short recovery after packet loss, or frame-accurate evidence, the GOP length and reference-frame policy belong in the release record.
- The visualization should state when users are seeing compressed media, sampled media, delayed media, or representative thumbnails.
Major section
Codec Decision Map
Codec selection starts with the evidence the application must preserve.
- Bandwidth and device compute then constrain what can be transported and decoded.
- Compatibility, metadata, and replay complete the route by asking whether future readers can interpret and investigate the record.
- The result is a defendable contract, not a codec popularity vote.
Major section
Common Defects
Container-codec confusion: assuming MP4 support means every video stream inside it will play.
- Over-compact payloads: using binary records without schema, versioning, or inspection tools.
- Lossy analysis damage: compressing signals in a way that removes features needed by analytics.
- Hidden delay: presenting delayed media as if it were live.
Deck summary
Key takeaways
The screen should show the gap or doubt.
- Latency means the time from an event to the result that needs it.: Measure it from capture to the visible chart or image.
- Visualization depends on data arriving in a form that can be decoded, trusted, aligned in time, and displayed.
- A dashboard may use JSON for sensor values and H.264 for a camera view at the same time.
- Storage must retain enough metadata for the decoder to reconstruct that meaning, and the visual layer must expose freshness and quality rather than only a number.
Retrieval practice
Recall check 1 of 3
Data Dora says: answer from memory, then check your reasoning.
Q1In IoT visualization, what is the difference between a codec and a container?
Show answer
Answer: B A codec encodes and decodes a signal, while a container is the wrapper that holds encoded media and metadata.
Retrieval practice
Recall check 2 of 3
Data Dora says: answer from memory, then check your reasoning.
Q2Why does Gorilla-style XOR compression shrink many real sensor value columns without losing exact values?
Show answer
Answer: A Columnar layouts let timestamp and numeric-value encoders exploit regularity while preserving exact raw values.
Retrieval practice
Recall check 3 of 3
Data Dora says: answer from memory, then check your reasoning.
Q3A dashboard shows a normal sensor value, but the data record does not include freshness or quality state. What is the main visualization risk?
Show answer
Answer: A Trustworthy visualization requires encoded metadata for timestamp, unit, identity, and data quality state.
Print reference
Answers
Answer key.
- B · A codec encodes and decodes a signal, while a container is the wrapper that holds encoded media and metadata.
- A · Columnar layouts let timestamp and numeric-value encoders exploit regularity while preserving exact raw values.
- A · Trustworthy visualization requires encoded metadata for timestamp, unit, identity, and data quality state.