2 Media Compression Mechanics: Audio, Image, and Video

2.1 Overview: Remove What the Receiver Cannot Use

Data Encoding and Codecs already establishes the decision layer: a media codec applies decomposition, quantization, and entropy coding, and a product team chooses among codecs by fidelity, latency, compute, compatibility, and replay needs. This chapter does not repeat that taxonomy. It opens the mechanism underneath it: how perception makes some audio error irrelevant, how an image block becomes spatial-frequency coefficients, and how a video encoder predicts motion before coding what remains.

The useful distinction is between two kinds of savings:

• Statistical redundancy means the source contains values that can be represented more compactly without changing the reconstruction.
• Perceptual irrelevancy means a changed reconstruction can still be indistinguishable, or acceptably close, for the intended listener or viewer.

Lossy media coding uses both. The risk is also in that sentence: “irrelevant” is never universal. It depends on the signal, the listener or viewing task, the playback path, and the encoder settings.

2.2 Practitioner: Turn an Image Block into an Ordered Coefficient Stream

JPEG’s transform mechanism starts from an $8\times8$ block of level-shifted pixel samples. For one component such as luminance, the two-dimensional discrete cosine transform computes one coefficient for every horizontal and vertical spatial-frequency pair $(u,v)$:

↔ Scroll the equation horizontally to inspect both cosine dimensions.

$\alpha(0)=\frac{1}{\sqrt{2}},\qquad \alpha(k)=1\text{ for }k>0.$

$C(0,0)$ is the DC coefficient: the block’s average level, scaled by the transform convention. The other 63 are AC coefficients describing progressively faster changes across the block. A smooth sky or wall usually concentrates energy near DC and the low-frequency corner. Sharp texture, lettering, or edges require more high-frequency coefficients.

For a constant block with every level-shifted sample equal to $a$, all cosine terms cancel except DC. Under the normalization above:

$C(0,0)=8a,\qquad C(u,v)=0\text{ for every }(u,v)\ne(0,0).$

That is energy compaction in its simplest form: 64 equal samples become one non-zero coefficient before quantization. Real blocks are not constant, but locally smooth image regions often move in the same direction.

2.3 Under the Hood: Predict Motion, Then Code the Error

Video adds a second source of redundancy: a block in the current frame often resembles a shifted block in a reference frame. A block matcher searches a bounded window for the candidate that minimizes a distortion measure. With sum of absolute differences (SAD):

$\operatorname{SAD}(\Delta x,\Delta y)=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}\left|T(i,j)-R(i+\Delta x,j+\Delta y)\right|.$

$T$ is the target block and $R$ is the reference frame. The selected displacement $(\Delta x,\Delta y)$ becomes a motion vector. Copying the matching reference block into the target position gives a prediction $P$; subtracting it from the target gives the residual:

$E(i,j)=T(i,j)-P(i,j).$

If the prediction is good, $E$ has smaller magnitudes and less structure than $T$. The encoder sends motion information plus a transformed, quantized, and entropy-coded residual. The decoder applies the same motion vector to its reference frame, decodes the residual, and adds the two. Prediction has changed what the transform must describe; it has not replaced the transform.

2.4 Summary

The chapter’s mechanisms form one argument about controlled loss. Psychoacoustic masking first raises a signal-dependent detection threshold, giving an audio coder a reason to place selected error where it is less audible. For images, an $8\times8$ DCT reorganizes a block into one DC and 63 AC spatial-frequency coefficients; quantization then creates high-frequency zeros, and JPEG zig-zag order clusters those zeros for run-length and entropy coding. Video adds prediction: motion estimation selects a reference displacement, motion compensation constructs the predicted block, and the residual preserves what that prediction missed.

Those savings are defensible only when the encoder names both the mechanism and its cost. A perceptual threshold can be exceeded, quantization discards detail, and a poor motion predictor can cost more than an intra-coded block. Treat every compression range as a measurement tied to particular content and settings, not as a timeless property of a codec. If the source units contradict the claimed reduction, reject the comparison rather than silently repairing it. That evidence discipline is what connects the audio, image, and video cases.

2.5 Key Takeaway

2.6 See Also

Continue according to the boundary you need to inspect next. Use Data Encoding and Codecs to step outward from transform and prediction mechanics to containers, GOP structure, and codec selection. Choose Voice Compression and Digital Telephony when the open question is speech coding, latency, header budget, or measured call quality. Audio Features for Machine Learning follows a different branch: its time-frequency representation supports classification rather than reconstructable playback. Finally, VoIP Client Architecture places these media operations inside the signaling, processing, packet, and device responsibilities of a complete calling product.

• Data Encoding and Codecs — the codec/container, selection, I/P/B-frame, GOP, and quality-evaluation decision layer this chapter extends.
• Voice Compression and Digital Telephony — PCM, companding, ADPCM/CELP/LPC, quality metrics, and VoIP packet-budget material intentionally not repeated here.
• Audio Features for Machine Learning — a different use of time-frequency decomposition: extracting classification features rather than reconstructable media.
• VoIP Client Architecture: Two Case Studies — how media processing and transport components assemble into a calling product.

Together, these routes separate three questions that should not be collapsed: how a representation saves bits, how the resulting payload behaves on a real-time path, and which product component owns each failure and item of evidence.