2 Media Compression Mechanics: Audio, Image, and Video

2.1 Overview: Remove What the Receiver Cannot Use

Data Encoding and Codecs already establishes the decision layer: a media codec applies decomposition, quantization, and entropy coding, and a product team chooses among codecs by fidelity, latency, compute, compatibility, and replay needs. This chapter does not repeat that taxonomy. It opens the mechanism underneath it: how perception makes some audio error irrelevant, how an image block becomes spatial-frequency coefficients, and how a video encoder predicts motion before coding what remains.

The useful distinction is between two kinds of savings:

• Statistical redundancy means the source contains values that can be represented more compactly without changing the reconstruction.
• Perceptual irrelevancy means a changed reconstruction can still be indistinguishable, or acceptably close, for the intended listener or viewer.

Lossy media coding uses both. The risk is also in that sentence: “irrelevant” is never universal. It depends on the signal, the listener or viewing task, the playback path, and the encoder settings.

2.2 Practitioner: Turn an Image Block into an Ordered Coefficient Stream

JPEG’s transform mechanism starts from an $8\times8$ block of level-shifted pixel samples. For one component such as luminance, the two-dimensional discrete cosine transform computes one coefficient for every horizontal and vertical spatial-frequency pair $(u,v)$:

$C(u,v)=\frac{\alpha(u)\alpha(v)}{4}\sum_{x=0}^{7}\sum_{y=0}^{7} f(x,y)\cos\!\left[\frac{(2x+1)u\pi}{16}\right]\cos\!\left[\frac{(2y+1)v\pi}{16}\right],$

$\alpha(0)=\frac{1}{\sqrt{2}},\qquad \alpha(k)=1\text{ for }k>0.$

$C(0,0)$ is the DC coefficient: the block’s average level, scaled by the transform convention. The other 63 are AC coefficients describing progressively faster changes across the block. A smooth sky or wall usually concentrates energy near DC and the low-frequency corner. Sharp texture, lettering, or edges require more high-frequency coefficients.

For a constant block with every level-shifted sample equal to $a$, all cosine terms cancel except DC. Under the normalization above:

$C(0,0)=8a,\qquad C(u,v)=0\text{ for every }(u,v)\ne(0,0).$

That is energy compaction in its simplest form: 64 equal samples become one non-zero coefficient before quantization. Real blocks are not constant, but locally smooth image regions often move in the same direction.

2.3 Under the Hood: Predict Motion, Then Code the Error

Video adds a second source of redundancy: a block in the current frame often resembles a shifted block in a reference frame. A block matcher searches a bounded window for the candidate that minimizes a distortion measure. With sum of absolute differences (SAD):

$\operatorname{SAD}(\Delta x,\Delta y)=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}\left|T(i,j)-R(i+\Delta x,j+\Delta y)\right|.$

$T$ is the target block and $R$ is the reference frame. The selected displacement $(\Delta x,\Delta y)$ becomes a motion vector. Copying the matching reference block into the target position gives a prediction $P$; subtracting it from the target gives the residual:

$E(i,j)=T(i,j)-P(i,j).$

If the prediction is good, $E$ has smaller magnitudes and less structure than $T$. The encoder sends motion information plus a transformed, quantized, and entropy-coded residual. The decoder applies the same motion vector to its reference frame, decodes the residual, and adds the two. Prediction has changed what the transform must describe; it has not replaced the transform.

2.4 Summary

• Psychoacoustic masking raises a signal-dependent detection threshold, allowing a perceptual coder to place some error where it is less audible.
• An $8\times8$ DCT expresses an image block as one DC and 63 AC spatial-frequency coefficients.
• Quantization creates many high-frequency zeros; JPEG zig-zag order clusters them for run-length and entropy coding.
• Motion estimation chooses a reference displacement, motion compensation builds a prediction, and the residual carries what the prediction missed.
• Compression ranges are measurements tied to content and settings, not timeless codec constants; inconsistent source units must be rejected rather than repaired invisibly.

2.5 Key Takeaway

2.6 See Also

• Data Encoding and Codecs — the codec/container, selection, I/P/B-frame, GOP, and quality-evaluation decision layer this chapter extends.
• Voice Compression and Digital Telephony — PCM, companding, ADPCM/CELP/LPC, quality metrics, and VoIP packet-budget material intentionally not repeated here.
• Audio Features for Machine Learning — a different use of time-frequency decomposition: extracting classification features rather than reconstructable media.
• VoIP Client Architecture: Two Case Studies — how media processing and transport components assemble into a calling product.