Cassandra
How Apache Cassandra works: partition and clustering keys, query-first modelling, the token ring, tunable consistency, LSM storage, tombstones and when not to use it.
On this page
The problem: more writes than one database can take
A chat app stores every message: 500 million a day, about 6,000 writes a second on average and 30,000 at peak, kept forever. A messaging inbox, an IoT platform with a million sensors reporting every 10 seconds, and an activity log all look the same: enormous write volume, data that grows without end, and reads that are almost always "the recent items for one thing".
A single PostgreSQL primary handles thousands to low tens of thousands of simple writes a second, and its disk fills up eventually. You could shard it yourself, but then your team owns routing, rebalancing, replication and failover for every shard.
Apache Cassandra is a distributed database built for exactly this. Every node is equal (there is no primary), data is spread across nodes by hashing a key, every piece is copied to several nodes, and adding capacity means adding machines. The price is that you give up joins, ad-hoc queries and, by default, strong consistency, and you must design your tables around the queries you will run.
The data model: partitions of sorted rows
Cassandra is usually called a wide-column store. In practice, with its query language CQL (which looks like SQL), you create tables with columns, and the important part is the primary key, which has two halves:
CREATE TABLE messages_by_channel (
channel_id bigint,
sent_at timeuuid,
sender_id bigint,
body text,
PRIMARY KEY ((channel_id), sent_at)
) WITH CLUSTERING ORDER BY (sent_at DESC);
- The partition key (
channel_id, inside the inner brackets) decides where the data lives. All rows with the same partition key form one partition, stored together on the same set of nodes. - The clustering key (
sent_at) decides the order of rows inside a partition. Rows are kept sorted on disk by it.
The partition key picks the nodes; the clustering key orders rows inside the partition
So the query the table was built for is cheap: it goes to one partition and reads a contiguous slice.
SELECT sender_id, body FROM messages_by_channel
WHERE channel_id = 42 LIMIT 50; -- newest 50, already in order
A query that does not give the partition key, such as WHERE sender_id = 7, would have to ask every node. Cassandra refuses it unless you add ALLOW FILTERING, which in production almost always means a full cluster scan.
Query-first modelling
In a relational database you model the entities and then write any query you like. In Cassandra you list the queries first and make one table per query, copying data between tables (denormalising) as needed. If you also need "all messages a user sent", you create messages_by_sender with PRIMARY KEY ((sender_id), sent_at) and write every message to both tables.
Two rules keep partitions healthy:
- Bound the partition size. Keep partitions under roughly 100 MB (tens of MB is better). A channel with ten years of messages breaks this, so add a time bucket to the partition key:
PRIMARY KEY ((channel_id, month), sent_at). Reads for "recent messages" touch the current month, and older months on demand. - Spread the load. A partition key with few values (a country, a status) creates hot partitions: one set of replicas does most of the work however many nodes you add. See hot keys and hot partitions.
The ring: how data is spread and copied
Each node owns ranges of a token ring: the partition key is hashed with Murmur3 to a 64-bit token, and the node whose range contains the token holds the first copy. With replication factor (RF) 3, the next two distinct nodes clockwise (in other racks or availability zones, if you configure that) hold the other copies. This is consistent hashing.
Each physical node owns many small ranges called virtual nodes (vnodes): 16 by default since Cassandra 4.0 (256 before). When a node joins, it takes a little data from many nodes instead of half of one neighbour's, and when one dies, many nodes share the work of rebuilding it.
Virtual nodes even out the ring, and a new server takes a little from everyone
Nodes learn about each other through gossip: every second, each node exchanges state with a few others, so the whole cluster learns quickly who is up. There is no leader and no single point of failure. A client driver knows the token map and sends each request straight to a replica, which acts as coordinator.
Tunable consistency
For every request, the client chooses a consistency level: how many replicas must answer before the coordinator replies.
| Level | Replicas that must answer (RF 3) | Use when |
|---|---|---|
ONE | 1 | Lowest latency, data where a stale read is fine (metrics, logs) |
QUORUM | 2 of 3 across all data centres | Read-your-writes with one region |
LOCAL_QUORUM | 2 of 3 in the local data centre | The usual choice with several regions: strong locally, no cross-region wait |
EACH_QUORUM | A quorum in every data centre (writes only) | Rarely; very slow |
ALL | 3 | Almost never: one down replica fails the request |
A quorum is a majority. The rule to remember: if R + W > N (replicas read plus replicas written is more than the replication factor), every read overlaps with the latest write on at least one replica.
QUORUM reads and writes overlap; ONE and ONE do not
Even with quorums, Cassandra is not linearizable for ordinary reads and writes. Conflicts are settled by last write wins: each cell carries a timestamp (usually set by the coordinator or client clock), and the highest timestamp wins. Clock skew between machines can make an older write "win". A write that failed to reach a quorum may still have reached one replica and appear later. Concurrent read-modify-write from two clients loses an update.
Cassandra heals divergence three ways: hinted handoff (a coordinator stores writes for a down replica, for up to 3 hours by default, and replays them when it returns), read repair (a read that sees disagreement fixes the stale copy) and anti-entropy repair (nodetool repair, or incremental repair, comparing data with Merkle trees). Regular repair is not optional, for the reason in the tombstone box below.
Lightweight transactions
When you really need "insert only if it does not exist" (a unique username, a seat), Cassandra offers lightweight transactions (LWT), which run the Paxos consensus protocol among the replicas:
INSERT INTO usernames (name, user_id) VALUES ('priya', 81) IF NOT EXISTS;
UPDATE seats SET holder = 81 WHERE show_id = 9 AND seat = 'C7' IF holder = null;
They are linearizable for that one partition, but cost several round trips between replicas (Paxos v2 in Cassandra 4.1 cut the number), so expect roughly 4–10× the latency of a normal write and much lower throughput under contention. Use them for rare, important decisions, never for every write. Multi-partition transactions (the Accord protocol) are under development for a future major release; today there are none.
Storage: why writes are fast and reads can be slow
Cassandra stores data in an LSM tree (log-structured merge tree): writes never update a file in place.
Commit log and memtable, flushed to immutable SSTables, merged by compaction
The compaction strategy is chosen per table and matters a great deal:
| Strategy | How it merges | Good for |
|---|---|---|
| Size-tiered (STCS) | Merges SSTables of similar size | Write-heavy tables; needs up to 2× free disk during big merges |
| Leveled (LCS) | Keeps levels of non-overlapping files | Read-heavy tables; a read touches few files; more write I/O |
| Time-window (TWCS) | One SSTable per time window, whole windows dropped on expiry | Time series with TTLs |
| Unified (UCS, 5.0) | A tunable mix of the above | New tables on 5.0, once you have tested it |
Tombstones
A delete cannot remove data from an immutable file, so Cassandra writes a tombstone: a marker saying "this cell or row was deleted at time T". Reads must read tombstones and skip the dead data they hide. Compaction finally drops a tombstone after gc_grace_seconds (10 days by default).
Performance numbers and a worked sizing
| Measure | Typical value |
|---|---|
| Write latency (LOCAL_QUORUM, same region) | ~1–5 ms at p99 on SSDs |
| Read latency, single partition | ~1–10 ms, more with many SSTables or tombstones |
| Writes per node | roughly 10,000–50,000 small writes/s, depending on hardware |
| Data per node | commonly 1–2 TB on SSD (more makes repair and rebuild slow) |
Worked example: the chat app. 30,000 message writes/s at peak, written to two tables (by channel and by sender), RF 3.
- Replica writes: 30 K × 2 tables × 3 copies = 180 K replica writes/s. At ~20 K per node, about 9 nodes; plan 12 for headroom and maintenance.
- Storage: 500 M messages/day × ~300 bytes × 2 tables × 3 copies ≈ 900 GB a day before compression (roughly half after). At ~1.5 TB per node, that is several new nodes a month, so either add nodes steadily or move old buckets to cheaper storage.
- Partition key:
(channel_id, month)keeps even the busiest channel's partitions bounded.
When to pick Cassandra, and when not
| Cassandra / ScyllaDB | DynamoDB | PostgreSQL | MongoDB | |
|---|---|---|---|---|
| Good at | Huge write volume, linear scaling, multi-region active-active | The same model, fully managed, pay per request | Transactions, joins, ad-hoc queries | Flexible documents, secondary indexes |
| Weak at | Ad-hoc queries, transactions, deletes, operational effort | Queries outside the key design; cost at steady high volume | Writes beyond one primary | Very high write volume at a lower cost per node |
| Use when | Write-heavy, key-based access, you run your own infrastructure | You are on AWS and want no operations | The data is relational or must be exactly right | Document-shaped data with varied queries |
ScyllaDB is a C++ reimplementation compatible with Cassandra's protocol, with higher throughput per node.
When not to use Cassandra: money and inventory that need multi-row transactions; data you will query in ways you cannot predict (analytics belongs in a warehouse); queues and frequently deleted data (tombstones); small datasets that fit one PostgreSQL server, where Cassandra's operational cost buys nothing; and any team without someone to own repairs, compaction and capacity planning.
In the interview
One million sensors each send a reading (sensor id, timestamp, temperature, battery) every 10 seconds. The app shows "readings for sensor X over the last 24 hours" and "the latest reading for sensor X". Data older than 90 days can go. Design the table(s), partition key, compaction and consistency.
Compare with our answer
readings_by_sensor with PRIMARY KEY ((sensor_id, day), ts) and CLUSTERING ORDER BY (ts DESC): 8,640 rows a day per sensor, a partition of well under 1 MB. "Last 24 hours" reads today's and yesterday's partitions; "latest reading" is LIMIT 1 on today's. Write with USING TTL 7776000 (90 days) and use TWCS with one-day windows, so whole SSTables expire without tombstone scans. Load: 100 K writes/s × RF 3 = 300 K replica writes/s, about 15–20 nodes. LOCAL_ONE writes are acceptable for sensor data; reads at LOCAL_ONE too, or LOCAL_QUORUM if the latest value must be fresh.
A strong answer mentions: a time bucket in the partition key · the partition size worked out · clustering order descending for "latest" · TTL plus TWCS instead of deletes · the write rate multiplied by RF · a deliberate consistency level.
A team moves its order system to Cassandra. Table orders has PRIMARY KEY ((status), created_at). The dashboard runs SELECT * FROM orders WHERE customer_id = ? ALLOW FILTERING. Stock is decremented by reading the count, subtracting in the app and writing it back at QUORUM. Cancelled orders are deleted. What will go wrong?
Compare with our answer
A strong answer mentions:
statusas the partition key gives a handful of giant, hot partitions ("pending" gets every new order); useorder_idor(customer_id)plus a bucket.ALLOW FILTERINGwithout the partition key scans the whole cluster; make anorders_by_customertable.- Read-modify-write on stock loses updates under concurrency even at
QUORUM(last write wins); use an LWT withIF stock = ?, or keep stock in a transactional database. - Deleting cancelled orders creates tombstones in already-huge partitions; prefer a status change or TTL, and run repair within
gc_grace_seconds. - Orders and stock need transactions across rows, which Cassandra does not have; this belongs in PostgreSQL, with Cassandra perhaps holding order history.
- Leaderless, peer-to-peer database: every node equal, gossip for membership, any node coordinates.
- Primary key = partition key (which nodes, via Murmur3 token ring) + clustering key (sort order inside the partition).
- Model one table per query; denormalise; no joins; queries without the partition key scan the cluster.
- Keep partitions bounded (under ~100 MB): add a time bucket. Avoid low-cardinality partition keys (hot partitions).
- RF 3, vnodes (16 by default). Consistency per request:
ONE,QUORUM,LOCAL_QUORUM; R + W > N for read-your-writes. - Conflicts: last write wins by timestamp; clock skew matters. LWT (Paxos) for compare-and-set on one partition, several times slower.
- LSM storage: commit log + memtable → SSTables → compaction (STCS, LCS, TWCS, UCS). Writes cheap, reads merge files.
- Deletes write tombstones; repair more often than
gc_grace_seconds(10 days) or deleted data comes back. - Not for transactions, ad-hoc queries, queues or small data.
1. A table is PRIMARY KEY ((user_id), event_time). Which query does Cassandra serve efficiently?
The partition key routes the query to one set of replicas, and the clustering key gives the order inside it. The other queries omit the partition key, so they would scan every node.
2. A cluster has RF 3. Writes use ONE and reads use ONE. A user updates their profile and immediately reloads it. What can happen?
With R + W = 2, which is not greater than N = 3, the read and write sets may not overlap. QUORUM or LOCAL_QUORUM on both sides gives read-your-writes.
3. A "pending jobs" table gets rows inserted and deleted constantly. After a few weeks, reads of the first 10 pending jobs time out. Why?
Deletes write tombstones, which reads must skip; with constant churn a read scans thousands of them before finding live rows. Queues belong in a message queue.
4. A node was offline for 14 days, with gc_grace_seconds at the default 10 days, and is restarted as it was. What is the risk?
Other replicas dropped the tombstones after 10 days; the returning node still has the old rows and repair spreads them back. Hints are kept for only 3 hours by default, so they do not cover it either.
5. Two users try to claim the username "priya" at the same moment. Which approach prevents both succeeding?
A check-then-write lets both clients see "free" before either writes, and last write wins silently. Only a lightweight transaction gives compare-and-set; secondary indexes do not enforce uniqueness.
6. A chat table uses PRIMARY KEY ((channel_id), sent_at). A few huge channels make reads slow and repairs fail. What is the best change?
Unbounded partitions are the problem; a bucket caps each partition's size. More replicas copy the giant partition, and LCS helps reads somewhat but leaves the partition unbounded.
7. Why are Cassandra writes cheap compared with an update in a B-tree database?
An LSM tree needs no read before write and no in-place update. The commit log is on disk for durability, and writes go to all replicas, not one.
Score 60% or more and the lesson is marked complete.