Technology deep dives · Chapter 27

Cassandra

How Apache Cassandra works: partition and clustering keys, query-first modelling, the token ring, tunable consistency, LSM storage, tombstones and when not to use it.

AdvancedInterview priority: Medium40 min
On this page
  1. The problem: more writes than one database can take
  2. The data model: partitions of sorted rows
  3. The ring: how data is spread and copied
  4. Tunable consistency
  5. Storage: why writes are fast and reads can be slow
  6. Performance numbers and a worked sizing
  7. When to pick Cassandra, and when not
  8. In the interview
  9. Quiz

The problem: more writes than one database can take

A chat app stores every message: 500 million a day, about 6,000 writes a second on average and 30,000 at peak, kept forever. A messaging inbox, an IoT platform with a million sensors reporting every 10 seconds, and an activity log all look the same: enormous write volume, data that grows without end, and reads that are almost always "the recent items for one thing".

A single PostgreSQL primary handles thousands to low tens of thousands of simple writes a second, and its disk fills up eventually. You could shard it yourself, but then your team owns routing, rebalancing, replication and failover for every shard.

Apache Cassandra is a distributed database built for exactly this. Every node is equal (there is no primary), data is spread across nodes by hashing a key, every piece is copied to several nodes, and adding capacity means adding machines. The price is that you give up joins, ad-hoc queries and, by default, strong consistency, and you must design your tables around the queries you will run.

The data model: partitions of sorted rows

Cassandra is usually called a wide-column store. In practice, with its query language CQL (which looks like SQL), you create tables with columns, and the important part is the primary key, which has two halves:

CREATE TABLE messages_by_channel (
    channel_id  bigint,
    sent_at     timeuuid,
    sender_id   bigint,
    body        text,
    PRIMARY KEY ((channel_id), sent_at)
) WITH CLUSTERING ORDER BY (sent_at DESC);
  • The partition key (channel_id, inside the inner brackets) decides where the data lives. All rows with the same partition key form one partition, stored together on the same set of nodes.
  • The clustering key (sent_at) decides the order of rows inside a partition. Rows are kept sorted on disk by it.

The partition key picks the nodes; the clustering key orders rows inside the partition

So the query the table was built for is cheap: it goes to one partition and reads a contiguous slice.

SELECT sender_id, body FROM messages_by_channel
WHERE channel_id = 42 LIMIT 50;                       -- newest 50, already in order

A query that does not give the partition key, such as WHERE sender_id = 7, would have to ask every node. Cassandra refuses it unless you add ALLOW FILTERING, which in production almost always means a full cluster scan.

Query-first modelling

In a relational database you model the entities and then write any query you like. In Cassandra you list the queries first and make one table per query, copying data between tables (denormalising) as needed. If you also need "all messages a user sent", you create messages_by_sender with PRIMARY KEY ((sender_id), sent_at) and write every message to both tables.

Two rules keep partitions healthy:

  • Bound the partition size. Keep partitions under roughly 100 MB (tens of MB is better). A channel with ten years of messages breaks this, so add a time bucket to the partition key: PRIMARY KEY ((channel_id, month), sent_at). Reads for "recent messages" touch the current month, and older months on demand.
  • Spread the load. A partition key with few values (a country, a status) creates hot partitions: one set of replicas does most of the work however many nodes you add. See hot keys and hot partitions.

The ring: how data is spread and copied

Each node owns ranges of a token ring: the partition key is hashed with Murmur3 to a 64-bit token, and the node whose range contains the token holds the first copy. With replication factor (RF) 3, the next two distinct nodes clockwise (in other racks or availability zones, if you configure that) hold the other copies. This is consistent hashing.

Each physical node owns many small ranges called virtual nodes (vnodes): 16 by default since Cassandra 4.0 (256 before). When a node joins, it takes a little data from many nodes instead of half of one neighbour's, and when one dies, many nodes share the work of rebuilding it.

Virtual nodes even out the ring, and a new server takes a little from everyone

Nodes learn about each other through gossip: every second, each node exchanges state with a few others, so the whole cluster learns quickly who is up. There is no leader and no single point of failure. A client driver knows the token map and sends each request straight to a replica, which acts as coordinator.

Tunable consistency

For every request, the client chooses a consistency level: how many replicas must answer before the coordinator replies.

LevelReplicas that must answer (RF 3)Use when
ONE1Lowest latency, data where a stale read is fine (metrics, logs)
QUORUM2 of 3 across all data centresRead-your-writes with one region
LOCAL_QUORUM2 of 3 in the local data centreThe usual choice with several regions: strong locally, no cross-region wait
EACH_QUORUMA quorum in every data centre (writes only)Rarely; very slow
ALL3Almost never: one down replica fails the request

A quorum is a majority. The rule to remember: if R + W > N (replicas read plus replicas written is more than the replication factor), every read overlaps with the latest write on at least one replica.

QUORUM reads and writes overlap; ONE and ONE do not

Even with quorums, Cassandra is not linearizable for ordinary reads and writes. Conflicts are settled by last write wins: each cell carries a timestamp (usually set by the coordinator or client clock), and the highest timestamp wins. Clock skew between machines can make an older write "win". A write that failed to reach a quorum may still have reached one replica and appear later. Concurrent read-modify-write from two clients loses an update.

Cassandra heals divergence three ways: hinted handoff (a coordinator stores writes for a down replica, for up to 3 hours by default, and replays them when it returns), read repair (a read that sees disagreement fixes the stale copy) and anti-entropy repair (nodetool repair, or incremental repair, comparing data with Merkle trees). Regular repair is not optional, for the reason in the tombstone box below.

Lightweight transactions

When you really need "insert only if it does not exist" (a unique username, a seat), Cassandra offers lightweight transactions (LWT), which run the Paxos consensus protocol among the replicas:

INSERT INTO usernames (name, user_id) VALUES ('priya', 81) IF NOT EXISTS;
UPDATE seats SET holder = 81 WHERE show_id = 9 AND seat = 'C7' IF holder = null;

They are linearizable for that one partition, but cost several round trips between replicas (Paxos v2 in Cassandra 4.1 cut the number), so expect roughly 4–10× the latency of a normal write and much lower throughput under contention. Use them for rare, important decisions, never for every write. Multi-partition transactions (the Accord protocol) are under development for a future major release; today there are none.

Storage: why writes are fast and reads can be slow

Cassandra stores data in an LSM tree (log-structured merge tree): writes never update a file in place.

Commit log and memtable, flushed to immutable SSTables, merged by compaction

The compaction strategy is chosen per table and matters a great deal:

StrategyHow it mergesGood for
Size-tiered (STCS)Merges SSTables of similar sizeWrite-heavy tables; needs up to 2× free disk during big merges
Leveled (LCS)Keeps levels of non-overlapping filesRead-heavy tables; a read touches few files; more write I/O
Time-window (TWCS)One SSTable per time window, whole windows dropped on expiryTime series with TTLs
Unified (UCS, 5.0)A tunable mix of the aboveNew tables on 5.0, once you have tested it

Tombstones

A delete cannot remove data from an immutable file, so Cassandra writes a tombstone: a marker saying "this cell or row was deleted at time T". Reads must read tombstones and skip the dead data they hide. Compaction finally drops a tombstone after gc_grace_seconds (10 days by default).

Performance numbers and a worked sizing

MeasureTypical value
Write latency (LOCAL_QUORUM, same region)~1–5 ms at p99 on SSDs
Read latency, single partition~1–10 ms, more with many SSTables or tombstones
Writes per noderoughly 10,000–50,000 small writes/s, depending on hardware
Data per nodecommonly 1–2 TB on SSD (more makes repair and rebuild slow)

Worked example: the chat app. 30,000 message writes/s at peak, written to two tables (by channel and by sender), RF 3.

  • Replica writes: 30 K × 2 tables × 3 copies = 180 K replica writes/s. At ~20 K per node, about 9 nodes; plan 12 for headroom and maintenance.
  • Storage: 500 M messages/day × ~300 bytes × 2 tables × 3 copies ≈ 900 GB a day before compression (roughly half after). At ~1.5 TB per node, that is several new nodes a month, so either add nodes steadily or move old buckets to cheaper storage.
  • Partition key: (channel_id, month) keeps even the busiest channel's partitions bounded.

When to pick Cassandra, and when not

Cassandra / ScyllaDBDynamoDBPostgreSQLMongoDB
Good atHuge write volume, linear scaling, multi-region active-activeThe same model, fully managed, pay per requestTransactions, joins, ad-hoc queriesFlexible documents, secondary indexes
Weak atAd-hoc queries, transactions, deletes, operational effortQueries outside the key design; cost at steady high volumeWrites beyond one primaryVery high write volume at a lower cost per node
Use whenWrite-heavy, key-based access, you run your own infrastructureYou are on AWS and want no operationsThe data is relational or must be exactly rightDocument-shaped data with varied queries

ScyllaDB is a C++ reimplementation compatible with Cassandra's protocol, with higher throughput per node.

When not to use Cassandra: money and inventory that need multi-row transactions; data you will query in ways you cannot predict (analytics belongs in a warehouse); queues and frequently deleted data (tombstones); small datasets that fit one PostgreSQL server, where Cassandra's operational cost buys nothing; and any team without someone to own repairs, compaction and capacity planning.

In the interview

Your turn: model sensor readings

One million sensors each send a reading (sensor id, timestamp, temperature, battery) every 10 seconds. The app shows "readings for sensor X over the last 24 hours" and "the latest reading for sensor X". Data older than 90 days can go. Design the table(s), partition key, compaction and consistency.

Compare with our answer

readings_by_sensor with PRIMARY KEY ((sensor_id, day), ts) and CLUSTERING ORDER BY (ts DESC): 8,640 rows a day per sensor, a partition of well under 1 MB. "Last 24 hours" reads today's and yesterday's partitions; "latest reading" is LIMIT 1 on today's. Write with USING TTL 7776000 (90 days) and use TWCS with one-day windows, so whole SSTables expire without tombstone scans. Load: 100 K writes/s × RF 3 = 300 K replica writes/s, about 15–20 nodes. LOCAL_ONE writes are acceptable for sensor data; reads at LOCAL_ONE too, or LOCAL_QUORUM if the latest value must be fresh.

A strong answer mentions: a time bucket in the partition key · the partition size worked out · clustering order descending for "latest" · TTL plus TWCS instead of deletes · the write rate multiplied by RF · a deliberate consistency level.

Your turn: spot the bugs

A team moves its order system to Cassandra. Table orders has PRIMARY KEY ((status), created_at). The dashboard runs SELECT * FROM orders WHERE customer_id = ? ALLOW FILTERING. Stock is decremented by reading the count, subtracting in the app and writing it back at QUORUM. Cancelled orders are deleted. What will go wrong?

Compare with our answer

A strong answer mentions:

  • status as the partition key gives a handful of giant, hot partitions ("pending" gets every new order); use order_id or (customer_id) plus a bucket.
  • ALLOW FILTERING without the partition key scans the whole cluster; make an orders_by_customer table.
  • Read-modify-write on stock loses updates under concurrency even at QUORUM (last write wins); use an LWT with IF stock = ?, or keep stock in a transactional database.
  • Deleting cancelled orders creates tombstones in already-huge partitions; prefer a status change or TTL, and run repair within gc_grace_seconds.
  • Orders and stock need transactions across rows, which Cassandra does not have; this belongs in PostgreSQL, with Cassandra perhaps holding order history.
Cassandra on one page
  • Leaderless, peer-to-peer database: every node equal, gossip for membership, any node coordinates.
  • Primary key = partition key (which nodes, via Murmur3 token ring) + clustering key (sort order inside the partition).
  • Model one table per query; denormalise; no joins; queries without the partition key scan the cluster.
  • Keep partitions bounded (under ~100 MB): add a time bucket. Avoid low-cardinality partition keys (hot partitions).
  • RF 3, vnodes (16 by default). Consistency per request: ONE, QUORUM, LOCAL_QUORUM; R + W > N for read-your-writes.
  • Conflicts: last write wins by timestamp; clock skew matters. LWT (Paxos) for compare-and-set on one partition, several times slower.
  • LSM storage: commit log + memtable → SSTables → compaction (STCS, LCS, TWCS, UCS). Writes cheap, reads merge files.
  • Deletes write tombstones; repair more often than gc_grace_seconds (10 days) or deleted data comes back.
  • Not for transactions, ad-hoc queries, queues or small data.
Check what stuck

1. A table is PRIMARY KEY ((user_id), event_time). Which query does Cassandra serve efficiently?

2. A cluster has RF 3. Writes use ONE and reads use ONE. A user updates their profile and immediately reloads it. What can happen?

3. A "pending jobs" table gets rows inserted and deleted constantly. After a few weeks, reads of the first 10 pending jobs time out. Why?

4. A node was offline for 14 days, with gc_grace_seconds at the default 10 days, and is restarted as it was. What is the risk?

5. Two users try to claim the username "priya" at the same moment. Which approach prevents both succeeding?

6. A chat table uses PRIMARY KEY ((channel_id), sent_at). A few huge channels make reads slow and repairs fail. What is the best change?

7. Why are Cassandra writes cheap compared with an update in a B-tree database?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.