Chapters

2 Media Compression Mechanics: Audio, Image, and Video

media-compression
data
audio-video

2.1 Overview: Remove What the Receiver Cannot Use

Data Encoding and Codecs introduces three codec steps: decomposition separates a signal into parts, quantization reduces numerical detail, and entropy coding packs common patterns. It also compares codecs by fidelity (how closely output matches input), latency (delay), compute cost, compatibility, and replay needs. This chapter explains those mechanisms: hearing can mask some audio errors; an image block can become spatial-frequency coefficients, values that describe image patterns; and a video encoder can predict motion before coding the remaining difference.

The useful distinction is between two kinds of savings:

  • Statistical redundancy means the source contains values that can be represented more compactly without changing the reconstruction.
  • Perceptual irrelevancy means a changed reconstruction can still be indistinguishable, or acceptably close, for the intended listener or viewer.

Lossy media coding uses both. The risk is also in that sentence: “irrelevant” is never universal. It depends on the signal, the listener or viewing task, the playback path, and the encoder settings.

2.1.1 Psychoacoustic Masking Is a Moving Threshold

The threshold in quiet is the minimum sound-pressure level a typical listener can detect at each frequency in a quiet environment. It is not flat: human hearing is most sensitive through part of the mid-frequency range and less sensitive at very low and very high frequencies. A strong tone or narrow band of energy raises that detection threshold around itself. Energy below the raised masking threshold is present physically but is less likely to be heard.

That is the mechanism a perceptual audio encoder exploits. It estimates energy across time and frequency, predicts which quantization noise would fall below the local masking threshold, and spends fewer bits where the resulting error is least audible. The encoder is not deleting “high frequencies” as a single rule. It is allocating error against a threshold that changes with the signal.

Quiet threshold

The baseline detectability curve varies with frequency even when no masker is present.

Masker

A strong component raises the nearby threshold; its effect spreads unevenly across frequency and time.

Noise allocation

Quantization error is shaped toward regions where the model predicts it will remain below that threshold.

The source lecture makes this audible with the 1990 Brandenburg and Johnston “Miracle 13dB” demonstration. Its four listening conditions are:

  1. A. Original

    The reference signal without added noise.

  2. B. White noise

    The original plus spectrally broad noise at -13 dB relative to the signal; the noise is obvious.

  3. C. Shaped noise

    The same nominal noise-to-signal relationship, but shaped against the signal's masking threshold; it can be difficult to hear.

  4. D. Difference

    The removed or added component heard on its own, proving that substantial energy can be perceptually hidden.

This page does not counterfeit the listening test. The source slide points to audio examples, but the estate does not contain licensed A/B/C/D assets for this demonstration. The registry-matched workbench below can generate tone-plus-noise and expose RMS and spectrum; it is a frequency-analysis prelude, not a claim that generic noise controls reproduce a validated masking experiment.

Audio sensing workbench narrated tour

Play silently with captions on provisional storyboard timing; narration audio is pending. Production pacing is independent of model time. Choose Reduced motion for a complete still at each beat.

Play the tour to open the workbench. Pause, replay a beat, or read the full transcript below. Situations the model cannot show use captions only and keep the existing frame.

Full transcript — Audio sensing workbench (8 beats)

B1 — Room sensor question
You are testing an internet-connected sound sensor for a room. Your job is to distinguish a steady tone from a brief change. The choice matters: an analysis setting that separates frequencies more finely also takes in a longer stretch of sound. How much time does finer detail cost?

Model caption: Hypothetical room-sensor test: finer frequency detail uses a longer span of sound.

B2 — Generated reference signal
The chapter describes an audio encoder estimating energy across time and frequency. Before studying compression, use a simpler experiment: a generated tone with light noise. Frequency analysis describes what frequencies are present. The workbench generates its own signal; it uses no microphone.

Model caption: Generated 480 Hz tone with light noise; no microphone. Frequency analysis precedes the compression lesson.

B3 — Windows and bins
Start with a tone at four hundred eighty hertz. Keep the sampling rate at forty-eight thousand samples per second. An analysis window is the span of samples used together. A frequency bin is a frequency slot in the resulting spectrum. Narrower bins give finer frequency detail.

Model caption: Rate: 48,000 samples/s. Window: samples analyzed together. Bin: a frequency slot.

B4 — 2048 sample calculation
Use two thousand forty-eight samples. Divide forty-eight thousand by two thousand forty-eight: each bin spans twenty-three point four three seven five hertz. For the window, divide two thousand forty-eight by forty-eight thousand, then multiply by one thousand to convert seconds to milliseconds. Rounded, that gives forty-two point seven milliseconds.

Model caption: 48,000 ÷ 2,048 = 23.4375 Hz. (2,048 ÷ 48,000) × 1,000 ≈ 42.7 ms.

B5 — 4096 sample comparison
Now use four thousand ninety-six samples, keeping the tone fixed. Divide forty-eight thousand by four thousand ninety-six: the bin width becomes eleven point seven one eight seven five hertz. Divide four thousand ninety-six by forty-eight thousand, then multiply by one thousand: the window becomes about eighty-five point three milliseconds. Half the bin width costs twice the time span.

Model caption: 48,000 ÷ 4,096 = 11.71875 Hz (display: 11.7188). (4,096 ÷ 48,000) × 1,000 ≈ 85.3 ms. Bin width halves; window doubles.

B6 — Change the tone
For your sensor test, change the generated tone from four hundred eighty to nine hundred sixty hertz. Its peak moves toward the higher frequency. The analysis window remains about eighty-five point three milliseconds, because changing the tone has not changed the sample count or sampling rate.

Model caption: Input tone: 960 Hz. Window: still 85.3 ms. Tone frequency changed; sample count and rate did not.

B7 — Longer is not always better
The misconception is that a longer analysis window is always better. It gives finer frequency detail, but it reacts more slowly to a short event. For a brief tone change, that extra time span matters alongside the narrower bins.

Model caption: Longer is not always better: 2,048 samples → 23.4375 Hz / 42.7 ms; 4,096 → 11.71875 Hz / 85.3 ms. Finer frequency detail reacts more slowly to a short event.

B8 — Choose the window
Choose the window for the change you need to follow, because finer frequency detail costs time.

Model caption: Choose the window for the change you need to follow, because finer frequency detail costs time.

2.2 Practitioner: Turn an Image Block into an Ordered Coefficient Stream

JPEG’s transform mechanism starts from an 8×88\times8 block of level-shifted pixel samples. For one component such as luminance, the two-dimensional discrete cosine transform computes one coefficient for every horizontal and vertical spatial-frequency pair (u,v)(u,v):

↔ Scroll the equation horizontally to inspect both cosine dimensions.

C(u,v)=α(u)α(v)4∑x=07∑y=07f(x,y)cos⁡ ⁣[(2x+1)uπ16]cos⁡ ⁣[(2y+1)vπ16],C(u,v)=\frac{\alpha(u)\alpha(v)}{4}\sum_{x=0}^{7}\sum_{y=0}^{7} f(x,y)\cos\!\left[\frac{(2x+1)u\pi}{16}\right]\cos\!\left[\frac{(2y+1)v\pi}{16}\right],

α(0)=12,α(k)=1 for k>0.\alpha(0)=\frac{1}{\sqrt{2}},\qquad \alpha(k)=1\text{ for }k>0.

C(0,0)C(0,0) is the DC coefficient: the block’s average level, scaled by the transform convention. The other 63 are AC coefficients describing progressively faster changes across the block. A smooth sky or wall usually concentrates energy near DC and the low-frequency corner. Sharp texture, lettering, or edges require more high-frequency coefficients.

For a constant block with every level-shifted sample equal to aa, all cosine terms cancel except DC. Under the normalization above:

C(0,0)=8a,C(u,v)=0 for every (u,v)≠(0,0).C(0,0)=8a,\qquad C(u,v)=0\text{ for every }(u,v)\ne(0,0).

That is energy compaction in its simplest form: 64 equal samples become one non-zero coefficient before quantization. Real blocks are not constant, but locally smooth image regions often move in the same direction.

2.2.1 The 8 by 8 Basis Map and Zig-Zag Order

The grid below labels each basis coordinate with its JPEG scan position. Horizontal spatial frequency increases across the columns; vertical spatial frequency increases down the rows. The scan starts at DC, crosses low-frequency diagonals, and reaches the highest-frequency corner last.

Quantization divides each coefficient by a position-dependent step and rounds the result:

C^(u,v)=round⁡ ⁣(C(u,v)Q(u,v)).\widehat{C}(u,v)=\operatorname{round}\!\left(\frac{C(u,v)}{Q(u,v)}\right).

Larger steps in less perceptually important high-frequency positions turn small coefficients into zeros. The zig-zag does not create compression by itself. It turns the two-dimensional coefficient block into a one-dimensional sequence that tends to put non-zero low-frequency values first and a long tail of zeros last. Run-length and entropy coding can then represent that zero tail cheaply.

2.3 Under the Hood: Predict Motion, Then Code the Error

Video adds a second source of redundancy: a block in the current frame often resembles a shifted block in a reference frame. A block matcher searches a bounded window for the candidate that minimizes a distortion measure. With sum of absolute differences (SAD):

SAD⁡(Δx,Δy)=∑i=0N−1∑j=0N−1∣T(i,j)−R(i+Δx,j+Δy)∣.\operatorname{SAD}(\Delta x,\Delta y)=\sum_{i=0}^{N-1}\sum_{j=0}^{N-1}\left|T(i,j)-R(i+\Delta x,j+\Delta y)\right|.

TT is the target block and RR is the reference frame. The selected displacement (Δx,Δy)(\Delta x,\Delta y) becomes a motion vector. Copying the matching reference block into the target position gives a prediction PP; subtracting it from the target gives the residual:

E(i,j)=T(i,j)−P(i,j).E(i,j)=T(i,j)-P(i,j).

If the prediction is good, EE has smaller magnitudes and less structure than TT. The encoder sends motion information plus a transformed, quantized, and entropy-coded residual. The decoder applies the same motion vector to its reference frame, decodes the residual, and adds the two. Prediction has changed what the transform must describe; it has not replaced the transform.

2.3.1 A Worked Decision: When the Residual Stops Being Cheap

A production encoder compares candidate modes; “inter” is not automatically cheaper merely because a previous frame exists.

  1. Data Dora compares one target image block with its best reference block from an earlier frame.

    Compare a target block with its best earlier-frame prediction.

  2. Dora subtracts them into a sparse difference grid with five small nonzero cells.

    Count how many difference samples remain and how large they are.

  3. Dora compares measured predicted-block and self-contained-block cost, choosing only the cheaper candidate without a universal winner badge.

    Choose the cheaper tested mode; an earlier frame does not guarantee a saving.

CP-0113 decision strip: Suppose a 4×44\times4 target block differs from its best reference candidate by these signed sample values:

The first panel compares a target block with its best candidate from an earlier frame. The second shows their difference: only five of sixteen residual samples are non-zero, and every magnitude is one. The matrix below gives the signed values. This prediction is a strong transform input. The last panel compares the cost of the predicted block with an intra-coded block. If a scene cut makes most residual magnitudes large, the motion vector plus residual can cost more than intra coding.

E=[00100−1100000100−1].E= \begin{bmatrix} 0&0&1&0\\ 0&-1&1&0\\ 0&0&0&0\\ 1&0&0&-1 \end{bmatrix}.

Data Encoding and Codecs owns the higher-level I/P/B-frame and GOP policy decision. This chapter supplies the lower mechanism: references produce predictions, predictions produce residuals, and residuals still traverse transform, quantization, and lossless packing. Real-Time Protocol Workflows owns the RTP/RTCP hand-off shown at the source diagram’s boundary; it is not re-taught here.

2.3.2 Read Source Rate Ranges as Evidence, Not Targets

The source’s final retained slide compares raw rates with example compressed ranges. These are lecture-era examples, not current codec recommendations. Their value is the scale of the mechanism’s payoff and the reminder that content and encoder settings control the result.

Media
Raw example
Source compressed range
Derived reduction
Wideband audio
224 kbps
8-12 kbps
about 19-28 times
CD audio
1.41 Mbps
64-256 kbps
about 5.5-22 times
Still image
3.14 MB
Rejected: slide prints 30-300 MB
No honest ratio from an internally inconsistent unit
720p raw video
530 Mbps
0.768-1.536 Mbps
about 345-690 times

Do not silently repair source data. The still-image row's printed compressed range is larger than its raw image and is labelled MBytes, so it fails its own compression claim. It may have intended kBytes, but this chapter does not guess. A product record should instead measure the actual images, encoder, quality setting, and decoder path.

2.4 Summary

Psychoacoustic masking raises the hearing threshold in a way that depends on the sound. An audio coder can place some error below that threshold, where it is less audible. For images, an 8×88\times8 DCT turns a block into one DC and 63 AC spatial-frequency coefficients. Quantization creates high-frequency zeros. JPEG zig-zag order groups those zeros for run-length and entropy coding. For video, motion estimation selects a shift in a reference frame. Motion compensation builds a predicted block, and the residual keeps what the prediction missed.

Each saving has a cost. Error can exceed a perceptual threshold, and quantization discards detail. A poor motion predictor can cost more than an intra-coded block. Compression ranges depend on the content and settings used to measure them; they are not fixed traits of a codec. If source units contradict a claimed reduction, reject the comparison instead of guessing a correction.

2.5 Key Takeaway

A perceptual threshold can hide some audio error. A transform can concentrate image energy, and a reference frame can predict video content. For each saving, state the mechanism, the error it adds, and the evidence that the intended receiver can still use the result.

2.6 See Also

Use Data Encoding and Codecs to step outward from transform and prediction mechanics to containers, GOP structure, and codec selection. Choose Voice Compression and Digital Telephony when the open question is speech coding, latency, header budget, or measured call quality. Audio Features for Machine Learning follows a different branch: its time-frequency representation supports classification rather than reconstructable playback. Finally, VoIP Client Architecture places these media operations inside the signaling, processing, packet, and device responsibilities of a complete calling product.

Together, these routes separate three questions that should not be collapsed: how a representation saves bits, how the resulting payload behaves on a real-time path, and which product component owns each failure and item of evidence.

Hi, I’m Broker Bex. What do you need?

Find help

Let’s find what you need

Find a lesson or return to your module guide.

Give feedback

Help improve this chapter

Tell us what helped or what needs fixing.

Tips

Try a small next step

A few ways to get more from this chapter.

Ask AI

Continue the conversation

Take the conversation further with your preferred assistant.

AI help is available in supported modules when enabled in Settings.

No live person answers on independent study. Feedback does not promise a personal reply.

For welfare or safety concerns, follow your school’s procedures outside IoTClass.