4 CAP Theorem and Database Categories

data-storage

cap

theorem

sharding

A storage link between two sites fails while an operator transfers a gateway to a new customer. The CAP theorem forces a choice about what the isolated replica may promise when it still names the old owner.

4.1 Start With the Broken Link

Choose What Each Record Does During a Split

Picture a building that loses its cloud link for twelve minutes. Room readings keep arriving at the edge, while two managers try to change the named owner of a door device from different places. The data owner must decide which work may continue and which must wait for one agreed answer.

Separate the records by consequence. A temperature sample can often be stored locally with source, event time, sequence, quality, and replay state. An ownership change may need to stop, queue for review, or use one named authority. Write the rule before the link fails.

Break the link, write on both sides, restart a node, repeat old work, change a clock, and reconnect under load. Check which reads are stale, which writes were refused, how conflicts are found, and how delayed samples are marked. Save a trace that another reviewer can replay.

Keep urgent local control safe while stores disagree. A distant copy can support history and fleet work, but it must not be the sole path for a time-bound building response. Do not call an accepted write correct until the record’s authority rule supports that claim.

This opening does not pick a database product or promise that one setting fits all data. Practitioner builds the record-by-record outage test. Under the Hood examines coordination, stale reads, conflict repair, queues, clocks, and recovery.

Use this split-state check:

Name each record class.
Name its one owner.
State what may wait.
State what may write.
State what must stop.
Cut the link once.
Write on both sides.
Repeat one old write.
Restart one store once.
Join the two sides.
Mark each stale read.
Prove the replay order.
Reopen when rules change.

CAP becomes real when a gateway loses the cloud link while devices keep working. Some records can wait, some can be accepted locally and replayed, and some must not split into two different truths. Review the behavior record by record before using the database label as a shortcut.

Data Dora

“Every reading has a time and a cost — decide retention before you decide the database.”

Here Dora tracks each record class through the partition: what keeps accepting writes, what waits, and what the replay trace must prove.

Overview: Review Partition Behavior By Record Class

CAP theorem is a way to reason about distributed storage when communication breaks between nodes that normally coordinate. The useful classroom version is not "pick CP or AP." It is "for this record class, what should the system do while parts of the system cannot talk?"

In IoT, partitions are normal operating conditions. Devices lose wireless links, gateways go offline, edge sites buffer locally, cloud regions fail over, and maintenance can isolate components. During that split, a storage design must either protect one current answer by coordinating, rejecting, or queueing work, or keep accepting work and later reconcile stale or conflicting state.

If you only need the intuition, use this rule: CAP is reviewed at the data-class boundary, not at the marketing category of the database product.

Before comparing record classes, use Figure 4.1 to see how the same edge-to-cloud partition creates different obligations for authority state and append-only telemetry.

CAP evidence timeline for a 12-minute edge-to-cloud outage comparing consistency-first owner-state handling with availability-first telemetry replay evidence. — Figure 4.1: CAP evidence timeline for a 12-minute edge-to-cloud outage.

Read Figure 4.1 from the link failure through the two record-class paths: owner state rejects or queues unsafe changes, while telemetry buffers locally and later proves ordering, deduplication, and delayed-arrival marking.

Read Figure 4.1 from the link failure through the two record-class paths. Owner state rejects or queues changes that could split authority, while telemetry can remain available only when its replay proves ordering, deduplication, and delayed-arrival marking. That contrast is the chapter’s running rule: partition behavior belongs to the record contract, not the database label.

Local building sensors keep sending records to an edge gateway while Data Dora observes its broken connection to the cloud. — CW-0005 walkthrough: record-class behavior during a partition keeps authority changes waiting while telemetry buffers with evidence, then requires ordered duplicate-safe replay and visible review.

Dora separates owner-change cards with person symbols from append-only sensor-reading cards before choosing either path's behavior. — CW-0005 walkthrough: record-class behavior during a partition keeps authority changes waiting while telemetry buffers with evidence, then requires ordered duplicate-safe replay and visible review.

Worked example: a building automation platform loses the link between an edge gateway and the cloud for 12 minutes. The telemetry table may keep accepting local temperature samples if each sample has a device id, event time, sequence key, quality flag, and replay rule. Late samples can then be appended after reconnect without pretending they were seen live. The release evidence is a replay trace that shows duplicates are ignored and delayed samples are marked as delayed.

The same outage is different for device ownership or firmware desired state. If an operator in the cloud assigns a new owner while the isolated site also accepts an owner change, the system may converge to the wrong authority. A defensible consistency-first path might reject owner changes while the authority service is unreachable, queue a signed request for later review, or require manual reconciliation before devices accept the new state. CAP review starts by separating those record classes.

Dora’s Retention Ledger

Reading: a building automation platform loses the edge-to-cloud link for 12 minutes while local temperature sampling continues.
Keep: a device id, event time, sequence key, quality flag, and replay rule on every sample.
Cost: the release evidence is a replay trace showing duplicates are ignored and delayed samples are marked as delayed.

The Three Terms In Release Language

Consistency

Reads reflect the latest accepted write under the system's consistency model. In CAP discussions this is closer to a single current answer than to schema validity.

Availability

A non-failing node returns a non-error response. That response may be stale or later reconciled if the design chooses availability during a partition.

Partition Tolerance

The system has an explicit behavior when nodes that should coordinate cannot communicate. In IoT, this behavior must be tested rather than assumed away.

IoT Examples

Compare these examples by consequence. Authority and firmware conflicts can create unsafe state, while append-only telemetry can often remain available if replay preserves identity, time, order, and quality. A latest-value cache is different again because it is derived and must expose freshness and rebuild rules.

Firmware desired state: conflicting accepted writes can update devices to the wrong target, so coordination, versioning, or rejection may be required.

Telemetry history: append-only samples may continue locally if timestamps, quality flags, idempotency, replay, and late-arrival rules are proven.

Device ownership: two partitions granting different owners is usually unsafe, so the authority boundary needs stricter behavior.

Latest dashboard value: a cache may continue answering if it is labeled as derived state with freshness and rebuild rules.

Overview Knowledge Check

Practitioner: Build A Partition Decision Ledger

A practical CAP review turns theory into a decision ledger. For each important data class, record the writer, source of truth, consequence of stale or conflicting data, chosen partition behavior, conflict policy, and evidence required before release.

This ledger prevents vague statements such as "the platform is AP" or "the database is strongly consistent." The same product may protect firmware state with coordination, accept telemetry locally with replay, and serve dashboard latest values from a cache. Each role needs its own partition decision.

Review Workflow

Move through this workflow in order so the release record stays auditable. Classification identifies the fact, consequence sets the risk, chosen behavior defines the partition response, reconciliation explains convergence, and attached tests prove the design behaves that way under an actual link failure.

Step 1, classify the record. Identify whether the data is telemetry, command, device registry, ownership, firmware state, event, cache, media artifact, or analytical output.

Step 2, rank the consequence. Describe what happens if the value is wrong, stale, missing, duplicated, reordered, or accepted in two places.

Step 3, choose behavior. Decide whether to reject, queue, accept locally, degrade to read-only, use stale reads, or allow derived answers.

Step 4, define reconciliation. Name the conflict resolver, idempotency key, replay order, timestamp source, audit trail, and manual escalation path.

Step 5, attach evidence. Keep partition tests, failover tests, replay tests, restore drills, stale-read labels, and owner approval with the release record.

Partition Decision Ledger

Record Class

Partition Risk

Likely Behavior

Evidence To Collect

Ownership and authorization

Two partitions may grant or revoke different authority for the same device.

Coordinate with a single authority, reject when authority is unavailable, or require explicit recovery workflow.

Authority boundary, conflict rejection, audit log, restore drill, and access-review approval.

Firmware desired state

Conflicting targets can push devices into unsafe or hard-to-recover states.

Use monotonic versions, approval gates, rollback rules, and no silent merge of incompatible targets.

Partition test, rollout dry run, rollback test, owner approval, and audit evidence.

Append-only telemetry

Samples may arrive late, duplicate, out of order, or with incomplete quality context.

Buffer locally, preserve source timestamps, replay idempotently, and mark late or uncertain readings.

Clock policy, sequence key, duplicate handling, late-arrival rule, quality propagation, and replay test.

Latest-value cache

Users may see stale derived state while durable history is unavailable or delayed.

Continue only with freshness labels, source-of-truth link, stale-read policy, and rebuild path.

Cache invalidation test, rebuild proof, freshness display, lag scenario, and dashboard owner approval.

Database Categories Are Shortcuts

Relational databases, wide-column stores, document databases, event logs, time-series databases, object stores, and caches can all be configured in ways that change partition behavior. Replication mode, quorum settings, read concern, write concern, client retries, batching, conflict handling, region topology, and operator runbooks matter more than the category label.

A release review should therefore treat category claims as hypotheses. If a team says a store is consistent enough, the review asks for the exact write path, read path, failure mode, and test evidence. If a team says a store is available enough, the review asks what users may read while data is stale and how conflicts converge after reconnect.

Practitioner Knowledge Check

Under The Hood: CAP Claims Depend On Mechanics

CAP is often reduced to labels, but the mechanics decide whether a claim holds. A system that coordinates before accepting a write needs leader election, quorum or consensus behavior, timeout handling, client retry semantics, and clear rejection behavior. A system that accepts work during a partition needs conflict detection, replay, idempotency, ordering, and a way to make stale or discarded state visible.

Quorum shorthand can help explain the intuition. If reads and writes must overlap across replicas, a later read can observe an accepted write under the assumptions of that system. The shorthand does not prove safe behavior by itself. The assumptions around clocks, failover, stale replicas, client retries, region latency, and application-level conflict handling still need evidence.

Worked example: a three-replica store requires two acknowledgements before accepting a firmware desired-state write. If one replica is isolated, the write can still be accepted by the two connected replicas, but a client timeout can make the operator retry. The release test must show whether the retry carries the same operation id, whether the store treats it as the same write, and whether the isolated replica receives the accepted version after reconnect. Without those mechanics, a "quorum write" label does not prove that devices see one target.

Dora’s Retention Ledger

Reading: a three-replica store needs two acknowledgements before accepting a firmware desired-state write.
Keep: whether a retried write carries the same operation id and is treated as the same write, not a second one.
Cost: without that check, a "quorum write" label does not prove the isolated replica ever converges to one target.

For availability-first telemetry, the mechanics are different. A gateway may buffer 720 one-second vibration readings during a 12-minute outage, then replay them when the link returns. The system needs a timestamp-source rule, a monotonically increasing sample id or idempotency key, a duplicate policy for resend after timeout, and a visible late-arrival marker. If the replay creates 730 records because 10 retries were counted twice, the system was available during the partition but not release-ready for that data class.

Mechanics That Change The Claim

Write Acceptance

The number and identity of acknowledgements, leader rules, retry behavior, batch boundaries, and timeout policy determine whether conflicting writes can be accepted.

Read Freshness

Read concern, replica selection, cache use, materialized views, dashboard polling, and freshness labels determine what users may see during or after a partition.

Replay And Repair

Sequence keys, idempotency keys, source timestamps, quality flags, compaction, merge rules, and audit records determine whether local acceptance can converge safely.

Operations Evidence

Partition drills, failover runs, restore tests, capacity limits, alert rules, runbooks, and owner sign-off determine whether the system can be trusted in production.

Failure Modes To Test

Exercise these failures as observable scenarios rather than labels. Interrupt coordination, repeat timed-out writes, serve a stale derived value, replay delayed telemetry, and recover the service. For each run, preserve timestamps, decisions, audit evidence, and the condition that blocks release when convergence is unproven.

Split authority: two sites accept different ownership, permission, firmware, or command state for the same device.

Retry amplification: clients retry writes after timeouts and create duplicates, stale overwrites, or out-of-order events.

Hidden stale reads: dashboards or APIs keep returning old derived values without visible freshness or source context.

Replay surprise: buffered telemetry reappears after reconnect but loses timestamps, quality flags, or duplicate handling.

Unproven recovery: a design promises convergence but has no partition drill, audit record, or restore evidence.

Under-the-Hood Knowledge Check

4.2 Sharding Strategies

4.2.1 Start With the Query That Must Stay Fast

Sharding is easiest to understand from the user’s query. A support page wants one device’s recent history now, while an operations report may need a whole fleet summary later. A good shard key makes the common path predictable and gives the expensive path a controlled plan instead of a surprise fan-out.

Overview: Sharding Is A Routing Contract

Sharding splits records across storage nodes so one machine is not responsible for every write, read, and byte of growth. The key design choice is the shard key. It decides where a record lands, which queries stay local, which queries fan out, and how painful growth or rebalancing will be later.

The useful question is not "is the database sharded?" The useful question is whether the routing contract matches the workload. IoT storage often needs recent device history, tenant isolation, fleet summaries, retention, replay, and restore. A shard key that helps one of those jobs can make another job expensive.

If you only need the intuition, use this rule: choose shard keys from required queries and write distribution, then prove the routing behavior before release.

Before selecting a shard key, use Figure 4.2 to compare the time, device, and hybrid routes against the queries that must remain local, the live writes that must stay balanced, and the retention work that must remain bounded. This preview makes routing a workload decision.

Before selecting a shard key, inspect the decision diagram in Figure 4.2 to compare the time, device, and hybrid routes against the queries that must remain local, the live writes that must stay balanced, and the retention work that must remain bounded. This preview makes routing a workload decision rather than a database feature label.

Sharding strategy decision guide branching on query pattern to time-based sharding, device-hash sharding, or a hybrid of both. — Figure 4.2: Sharding strategy decision guide for time, device-hash, and hybrid routing.

Read Figure 4.2 from the dominant query pattern into the candidate route, then test its write distribution, lifecycle boundary, and known fan-out path before accepting it.

Read Figure 4.2 from the dominant query pattern into the candidate route. Time routing supports bounded lifecycle work but can heat the current bucket; device routing preserves device locality but scatters fleet analytics; a hybrid must prove both dimensions. The decision therefore feeds the chapter’s routing contract and its release evidence.

Worked example: 12,000 devices each report every 30 seconds. That is 12,000 × 2 × 60 × 24 = 34,560,000 readings per day. A time-only shard key can make retention easy, but if every current reading lands in today's shard, one owner absorbs the entire live write load. A device-hash route across 8 shards would average about 4,320,000 readings per shard per day, but it makes a fleet-wide hourly average touch all 8 shards unless the system maintains summaries.

The shard-key review must therefore name both read paths. A support page that reads one device for the last 24 hours should route to one shard or a small known set of time chunks. A compliance report that reads every device in a region may intentionally scatter, but then it needs a fan-out budget, merge timeout, and summary fallback. The design is not proven until those local and scatter paths are tested with realistic counts.

Locality

Common reads should touch one shard or a small known shard set. Device, tenant, site, and time fields shape that locality.

Distribution

Writes should spread across enough shards to avoid a hot current partition, hot tenant, or hot device overwhelming one node.

Lifecycle

Retention, compaction, archival, and restore need bounded partitions or chunks that can be moved, verified, and removed safely.

Operations

Routing tests, skew dashboards, scatter budgets, rebalance drills, and backup evidence decide whether the sharding plan is ready.

Practitioner: Build The Shard-Key Ledger

A shard-key ledger forces the team to review locality, distribution, lifecycle, and operations before committing to a routing scheme. It should list each required query, the fields available in that query, the expected write distribution, and the evidence that proves routing under realistic load.

Do not treat partitioning, sharding, replication, and read replicas as interchangeable. Partitioning splits one logical table or dataset into physical pieces. Sharding assigns records to different ownership groups or nodes. Replication copies data for durability or read scale. Read replicas can help dashboards but do not fix a bad write-routing key.

Key Pattern

Works Well When

Common Failure

Evidence To Keep

Device or asset key

Most queries ask for one device, one asset, or a small known set of devices over a time window.

Fleet-wide analytics scatter across many shards unless summaries or analytics paths exist.

Single-device latency, targeted fan-out tests, per-shard write histogram, and summary-table plan.

Tenant or site key

Isolation, billing, access control, geography, or operational ownership matters.

A large tenant or site dominates its shard and becomes a noisy neighbor.

Tenant load distribution, move plan, quota policy, noisy-neighbor alarms, and restore drill.

Time bucket

Retention, chunk pruning, archival, and restore are dominated by time windows.

Current writes overload the newest bucket when time is the only routing field.

Current-bucket write rate, retention dry run, chunk size limits, and time-bound query plan.

Composite key

The workload needs both bounded time windows and locality for device, tenant, site, or region.

Too many dimensions make routing, backfills, and support work hard to explain and test.

Routing fixtures, key examples, boundary tests, rebalance drill, and support runbook.

Shard-Key Review Record

Primary query patterns:
Write distribution assumption:
Shard key:
Time or lifecycle boundary:
Queries that stay local:
Queries that fan out:
Hot-key mitigation:
Rebalance method:
Backup and restore evidence:
Release dashboard:

Under The Hood: Rebalancing And Hot Keys Decide Production Risk

A sharded design changes over time. Device fleets grow unevenly, tenants differ in write volume, regions add sites, retention windows change, and backfills compete with live ingestion. The routing contract therefore needs a rebalancing story, not only a first-day shard key.

Naive routing can make growth dangerous. If ownership is based only on a direct modulo of the current node count, adding or removing nodes may move many keys at once. More controlled approaches use stable ownership maps, token ranges, virtual buckets, explicit routing tables, or managed service mechanisms that move a bounded portion of the key space with verification.

Worked example: a telemetry cluster maps 1,024 virtual buckets onto 8 physical shards, so each shard owns about 128 buckets. If a ninth shard is added, the team can move 64 buckets first, which is 64 ÷ 1,024 = 6.25% of the key space, watch copy lag and read correctness, then continue in bounded waves. The release drill should record which buckets moved, how many device series were copied, how many writes arrived during dual-write or cutover, and what rollback condition would stop the move.

Hot keys need the same evidence. If one tenant sends 18,000 writes per minute while the median tenant sends 600, a tenant-only shard key can overload one owner even though the global average looks safe. The mitigation might be tenant-plus-device sub-buckets, quota enforcement, precomputed summaries, or moving that tenant to a larger shard group. Under the hood, the router must keep those exceptions deterministic so retries, backfills, and restore jobs use the same ownership map.

Mechanics To Prove

Routing Determinism

The same key must route predictably across application instances, deployments, and retries. Built-in process-randomized hashes are not suitable for durable routing.

Hot-Key Handling

High-rate devices, tenants, or sites may need controlled sub-buckets, quotas, summaries, or an explicit move plan.

Scatter Budgets

Queries that touch many shards need fan-out limits, timeout behavior, merge-cost evidence, and summary alternatives.

Move Safety

Rebalancing needs copy checks, dual-read or cutover rules, write ordering, rollback conditions, and restore validation.

Release Evidence

Collect release evidence in the same order that data moves: prove deterministic routing, observe distribution and hot keys, replay a bounded backfill, move a controlled ownership range, and test a shard failure. Together these records connect the initial key choice to safe growth and recovery.

Routing fixtures: known device, tenant, site, and time-boundary keys route to expected shards.

Skew dashboard: per-shard writes, bytes, queue depth, latency, error rate, and hot-key alerts are visible.

Backfill drill: historical replay is throttled and does not starve live writes.

Rebalance drill: a bounded key range moves while reads, writes, and restore checks remain correct.

Failure policy: the coordinator, router, or client behavior is known when one shard is slow or unavailable.

4.2.2 Summary

Sharding is a routing contract between records, queries, storage owners, and operations. The shard key should be selected from required queries, write distribution, lifecycle boundaries, and isolation needs. Time buckets help retention but can create hot current partitions when used alone. Device, tenant, site, and composite keys can improve locality but may make fleet analytics scatter without summaries. Production readiness depends on routing tests, skew dashboards, scatter budgets, rebalance drills, backup/restore evidence, and clear failure behavior.

Key Takeaway

Shard keys solve scale only when they match the workload and come with evidence for routing, skew, rebalancing, retention, and recovery.

4.2.3 See Also

Database Selection Framework

Choose storage roles by matching data shape, query evidence, consistency, lifecycle, and operations.

CAP Theorem and Database Categories

Review distributed behavior when partitions, replicas, and conflict policies matter.

Data Quality Monitoring

Keep validation, quarantine, and quality metrics visible across shards and replay paths.

Time-Series Database Fundamentals

Connect time buckets, append workloads, query paths, retention, and compression to shard design.

4.3 Keep an Ownership Change out of a Split Replica

Let both replicas initially hold ownership version 7, and let site A accept version 8 while site B cannot hear it. The theorem’s consistency requirement means that a successful authoritative read cannot present the old ownership as if it followed the completed change.

The theorem’s availability requirement concerns requests received by non-failing nodes, not a percentage on a yearly uptime dashboard. If the isolated site must always return a successful answer from its local copy, that answer can disagree with the latest completed ownership change.

A consistency-first design instead delays or refuses an answer when it cannot establish the required authority. The theorem explains why this response gives up availability for that operation during the partition, even though the local database process is still running.

This is not a requirement to stop every kind of storage on the device. The theorem applies to the guarantees being promised, so an append-only measurement queue can have a different contract from the record that authorizes account access.

Suppose the gateway samples at 5 s intervals during a 120 s outage, with its first queued record at 5 s. It accumulates 120 divided by 5 = 24 records; at an illustrative 80 bytes each, those records occupy 1,920 bytes before storage overhead.

That queue preserves local evidence without claiming that the distant database has committed it. The theorem is not contradicted by useful degraded operation: the designer has bounded the promise instead of providing both incompatible guarantees for the same authoritative operation.

After the link returns, replay measurements by stable identity and retain their event times. An older ownership update must not win merely because one replica has a fast clock; the authority rule needs stronger justification than arrival order.

Predict whether adding a third replica removes the theorem’s constraint. It can improve fault tolerance, but a partition may still separate the parties needed to decide which ownership state is authoritative.

Also predict whether the 24 buffered records made the live dashboard current throughout the outage. They repair history later; they could not provide timely remote observations while the link was absent.

The theorem does not prescribe a conflict-resolution algorithm; choosing how ownership disputes are settled after reconnection remains a separate design task. Under the theorem’s consistency requirement, synchronized clocks alone cannot reveal a completed write that the isolated replica has never received. The theorem also leaves capacity engineering open: additional shards can distribute queries, but they cannot carry coordination traffic across an unavailable link. Finally, the theorem’s availability condition is stronger than eventual recovery, because a delayed successful answer after reconnection does not make the earlier request available during the partition.

The theorem therefore matters at the data-class boundary: ownership, configuration and measurement history do not necessarily tolerate the same failure response. Read the chapter’s partition path as a decision about authority, then use its sharding guide for the separate question of where queries and records are placed; balanced storage alone cannot settle conflicting claims about a device’s owner.

4.4 Summary

CAP is a partition-behavior review for distributed storage, not a database popularity label. Review CAP decisions per IoT data class because firmware, ownership, telemetry, events, and derived views have different failure consequences. Consistency-first behavior coordinates, rejects, queues, or degrades when authority is unavailable. Availability-first behavior can be defensible for some append-only or derived data only when replay, conflict, freshness, and audit evidence are explicit. Database categories are useful shortcuts, but release evidence depends on configuration, client behavior, topology, operations, and tested failure modes.