Completeness Costs Latency and Memory
Completeness Costs Latency and Memory
Ada re-derives this chapter’s own numbers step by step, at full precision
ADA · CALCULATION AUDIT
Completeness Costs Latency and Memory
A stream-processing line emits 200 candidate A events per second, and each partial match can wait up to 30 seconds for its matching B event, so the engine may need to track up to 6,000 pending A records for that one rule before expiration. The chapter notes that doubling the window doubles the worst-case state, and that an unbounded pattern is effectively a memory leak. This audit asks the question that raises: does widening the watermark’s allowed lateness to catch stragglers cost only latency, or does it also inflate the pending-match memory the window-doubling example already suggests?
Companion to the chapter Lab: Advanced CEP — every number here comes from that chapter.
See the relationship before changing it
The figure reads from left to right. The blue card is match window. The middle card applies this page's rule. The green card is pending a records. Walk the arrows once: set the input, apply the rule, then read the result with its unit.
The retained audit below checks several chapter fixtures. This model keeps those stated values fixed and changes only match window, so the numeric fixture does not switch without explanation.
Derive the baseline in four named moves
- 1
Name the input. The chapter baseline is 30 s.
- 2
Name the relationship. pending records = 200 A events/s x match window
- 3
Substitute with units. 200 events/s x 30 s = 6,000 records
- 4
Read the result. Keep the unit beside the value. Use it only inside the technical boundary on this page.
Predict, then change match window
Try Predict the direction of pending records = 200 A events/s x match window. Test another match window, then compare pending a records.
Observe A longer match window catches later partners but holds more incomplete records in memory. Reset match window to 30 and compare pending a records.
Explain A longer match window catches later partners but holds more incomplete records in memory.
Check yourself
What should you do before trusting a moved-control result?
What does this small model leave out?
Treat candidate A at 200 events/s, a 30 s pattern window, and 30 s allowed lateness as the fixed audit case; predict the retention horizon.
Pending state grows from 6000 records at 30 s to 12000 at 60 s; removing the time bound makes the record count unbounded.
Every unmatched A occupies state until its match or expiry, so increasing the completeness horizon linearly increases both decision latency and retained records.
Ada: The under-the-hood layer gives one worked state figure: a line emitting 200 candidate A events per second, each waiting up to 30 seconds for its B, needs up to 6,000 pending records. Let me check it, then push on the “doubling the window doubles the state” claim, because it hides a third cost the latency discussion does not name.
Worst-case pending state is just rate times retention:
200 events/s x 30 s = 6,000 pending A records- Double the window:
200 x 60 = 12,000, exactly double, as claimed - Unbounded window:
200 x (no limit)grows without bound, which is why “A eventually followed by B” is a memory leak, not a semantics quibble
Here is the connection the chapter sets up but does not close. The watermark’s allowed lateness, the completeness knob, is retention time. Raising it to catch stragglers lengthens the effective window, so a 30 s pattern with 30 s of allowed lateness holds partial matches for 30 + 30 = 60 s, i.e. 200 x 60 = 12,000 records. Turning the completeness knob therefore does not buy you latency alone; it also linearly inflates memory. The design-meaning: the latency-versus-completeness curve is really a three-way budget of timeliness, completeness, and pending-state memory, and the only lever that bounds the third is a finite window, which is why bounding it is about survival, not taste.
Every number above is taken from the chapter’s own material and re-derived step by step.
Technical boundaries: This worst-case rate-times-retention model assumes a steady 200 events/s and no early matches; it excludes record size, key skew, garbage collection, checkpoint copies, backpressure, failures, and out-of-order bursts.