Technology deep dives · Chapter 29

Elasticsearch

How Elasticsearch works: inverted index and analyzers, shards and replicas, near-real-time refresh, BM25, aggregations, syncing from your database, and its limits.

AdvancedInterview priority: Medium40 min
On this page
  1. The problem: search that a database cannot do well
  2. Documents, mappings and analyzers
  3. Shards, replicas and the cluster
  4. Near real time: refresh, translog and segments
  5. Relevance: BM25
  6. Queries at scale: paging and aggregations
  7. Keeping it in sync with the database
  8. Performance numbers
  9. When to pick Elasticsearch, and when not
  10. In the interview
  11. Quiz

The problem: search that a database cannot do well

A shop has 50 million products. Users type "red runing shoes for men" and expect relevant results in under 200 ms, with typos forgiven, the best matches first, and counts by brand and price range down the side. In PostgreSQL, WHERE title ILIKE '%red%' AND title ILIKE '%shoes%' scans every row, cannot rank, cannot fix "runing", and computing the brand counts is another full pass.

Elasticsearch is a distributed search and analytics engine built on the Apache Lucene library. It stores JSON documents in an inverted index, which makes "documents containing these words" fast; ranks matches by relevance; computes counts and statistics (aggregations) over the matches; and spreads all of it over a cluster of machines.

The search lesson introduces inverted indexes and ranking. This lesson is about how Elasticsearch implements them, what it guarantees, and why it should almost never be your only copy of the data.

Documents, mappings and analyzers

An index (roughly, a table) holds JSON documents, each with an _id. A mapping (the schema) gives each field a type, and the type decides how it is indexed:

Field typeIndexed asUse for
textAnalysed into terms (words) in an inverted indexFull-text search: titles, descriptions
keywordOne exact term, plus columnar doc valuesFilters, sorting and aggregations: brand, status, tags
long, double, dateNumeric trees (BKD) plus doc valuesRanges, sorting, histograms
geo_point, dense_vectorGeo and vector (HNSW graph) structures"Within 5 km", semantic or kNN search

A field is often mapped both ways: title as text for searching and title.raw as keyword for sorting.

An analyzer turns text into terms, at index time and again at query time, so both sides agree:

"Running Shoes, RED!"
  → tokenizer:     [Running] [Shoes] [RED]
  → lowercase:     [running] [shoes] [red]
  → stemmer:       [run] [shoe] [red]

The inverted index then maps each term to the list of documents containing it (its postings list), with positions and frequencies. A query for "red shoes" is analysed the same way and becomes "documents in the postings lists of red and shoe". Typos are handled by fuzzy queries (terms within an edit distance of 1–2), and synonyms by a synonym filter in the analyzer.

Shards, replicas and the cluster

An index is split into primary shards, each a complete Lucene index holding a slice of the documents (chosen by hashing the routing value, by default the _id). Each primary has zero or more replica shards on other nodes.

Shards split the data; replicas copy it; queries scatter and gather

  • Writes go to the primary shard, which forwards them to its replicas in parallel and acknowledges once the in-sync copies have them.
  • Searches go to one copy of every shard (primary or replica), so replicas add query capacity as well as redundancy.
  • The number of primary shards is fixed when the index is created (the default is 1). Changing it means _split, _shrink or a full reindex. The replica count can change at any time.

A cluster elects one master node (a dedicated set of three master-eligible nodes is the norm) that manages the cluster state: mappings, which shard lives where, and node membership. Since 7.0 master election uses a quorum-based protocol, which removed the old split-brain configuration trap.

Aim for shards of roughly 10–50 GB. Too few, huge shards are slow to move and recover; too many tiny shards waste memory and make every search fan out to more pieces.

Near real time: refresh, translog and segments

Lucene's segments are immutable, so new documents are not searchable the instant they are written.

Durable in the translog at once; searchable after the next refresh

The default refresh_interval is 1 second. That is the "near real time" in the documentation: a user who saves a product and immediately searches may not see it. Options when that matters: show the saved item from the response rather than re-searching; index with ?refresh=wait_for (the request returns after the next refresh, without forcing one); or, for bulk loads, set the interval to 30 s or -1 during the load to make indexing much faster.

Segments are merged in the background; flush commits them

An update is a delete plus a new document: the old version is marked deleted and disappears at the next merge. Heavy update traffic means heavy merging, so a field that changes every second (a view count, live stock) is a poor fit for the main document.

Search-idle shards and the translog setting

If a shard has had no search for 30 seconds (index.search.idle.after), scheduled refreshes are skipped until a search arrives; that first search then waits for a refresh. By default, the translog is fsynced at the end of every index or bulk request (index.translog.durability: request), so an acknowledged write survives a node crash. Setting it to async syncs every 5 seconds instead: faster, but a crash can lose acknowledged writes.

Relevance: BM25

By default, Elasticsearch ranks matches with BM25, a refinement of TF-IDF. For each query term in a document:

  • Term frequency (TF): more occurrences score higher, but with diminishing returns (the parameter k1, 1.2 by default, controls how quickly it saturates). The tenth "shoe" adds little.
  • Inverse document frequency (IDF): rare terms count more. "Vegan" in a shoe catalogue is more telling than "shoe".
  • Length normalisation: a match in a short title beats one in a 2,000-word description (b, 0.75 by default).

The document's score is the sum over query terms. In practice you combine it with business signals: field boosts (title^3), function_score for popularity or recency, or a learning-to-rank model on the top few hundred results. Note that IDF is computed per shard, so with tiny indexes and several shards, scores can vary slightly by shard.

Queries at scale: paging and aggregations

Query then fetch, and why deep pages are expensive

Queries mix query context (how well does it match: scored) and filter context (does it match at all: yes or no, cached, cheap). Put exact conditions like brand = nike and price < 100 in filters.

Aggregations compute summaries over the matching documents, using doc values (a column-oriented copy of each field):

{
  "query": { "bool": {
    "must":   [{ "match": { "title": "red running shoes" } }],
    "filter": [{ "range": { "price": { "lte": 100 } } }] } },
  "aggs": {
    "brands": { "terms": { "field": "brand", "size": 10 } },
    "prices": { "histogram": { "field": "price", "interval": 25 } }
  }
}

Two accuracy limits worth knowing: a terms aggregation across shards is approximate (each shard returns its own top terms, so a term that is tenth everywhere can be missed; raising shard_size reduces the error), and cardinality (distinct count) uses HyperLogLog++, accurate to within a few per cent at large counts.

Keeping it in sync with the database

Elasticsearch is usually a secondary index: the database is the system of record, and the search index is a copy shaped for queries.

Change data capture feeds the indexer; the database stays the source of truth

Ways to keep the copy current:

ApproachHowCatch
Dual write from the appWrite to the database, then to ElasticsearchA crash between the two leaves them different forever; no retry log
Outbox or CDC (recommended)Read the database's change log (Debezium on the PostgreSQL WAL, MySQL binlog) into Kafka; an indexer applies itMore moving parts; seconds of lag
Periodic batchReindex rows changed since the last runMinutes of staleness; deletes are easy to miss

Why it is not a system of record

  • No multi-document transactions, and no foreign keys or unique constraints except on _id.
  • Mapping changes to existing fields need a reindex, which needs the source data.
  • Writes become visible only after refresh, and replicas can briefly disagree.
  • Its consistency and data-loss record has improved greatly (sequence numbers and the new cluster coordination since 6.x and 7.x), but its design goal is search, not being the last copy of money or orders.

Performance numbers

MeasureTypical value
Search latency~10–100 ms for typical queries; aggregations over many documents take longer
Indexing per data noderoughly 10,000–50,000 small documents/s with bulk requests
Shard size10–50 GB
MemoryJVM heap up to ~30 GB and about half of RAM; the rest left to the OS file cache, which Lucene relies on
Visibility delay~1 s (refresh interval)

Worked example: the shop. 50 M products at ~2 KB of indexed data each ≈ 100 GB. With 4 primary shards of ~25 GB and 1 replica, the cluster holds 200 GB. 2,000 searches/s at peak: every search touches all 4 shards, so each shard sees all 2,000. If one shard copy serves ~700–1,000 of these queries a second, each shard needs 3 copies: 1 primary and 2 replicas, 12 shard copies on 6 nodes. Measure with real queries before trusting that per-copy figure. Indexing is 500 updates/s, which is trivial. Prices and stock that change every few seconds are read from the database or a cache for the final page, not trusted from the index.

When to pick Elasticsearch, and when not

Elasticsearch / OpenSearchPostgreSQL full-textTypesense / MeilisearchManaged (Algolia)
Good atRelevance, facets, huge indexes, logs and analytics, vectorsSimple search next to your data, transactional consistencyEasy typo-tolerant site searchInstant search with no operations
Weak atOperational load, being the source of truthRanking quality, facets at scaleVery large or complex deploymentsCost at volume, less control
Use whenSearch or log analytics is a core featureTens of millions of rows and basic needsSmall to medium cataloguesBudget over control

OpenSearch is the fork AWS started in 2021 after Elastic changed its licence; Elastic added the AGPL as an option in 2024. Both are fine interview answers.

When not to use Elasticsearch: as the primary database; for exact lookups by ID (a database or cache is simpler); for data that changes many times a second per document; for strongly consistent reads ("did my payment go through?"); and when PostgreSQL full-text search (tsvector with a GIN index) already meets your needs.

In the interview

Your turn: log search

A platform produces 2 TB of application logs a day. Engineers search the last 7 days by service, level and free text, and keep 30 days for audits. Design the indexes, shard sizing and retention, and say what you would do with logs older than 7 days.

Compare with our answer

Use time-based indexes (one per day, or data streams with rollover at ~50 GB per primary shard), so retention is "delete yesterday's index", never deleting documents. 2 TB/day of raw logs is often somewhat less indexed (compression, mapping only searchable fields as text, the rest as keyword or not indexed); assume ~2 TB/day ≈ 40 primary shards of 50 GB per day. Keep 7 days on fast "hot" nodes with 1 replica (~28 TB), then move to cheaper "warm/cold" nodes or searchable snapshots in object storage for days 8–30, managed by index lifecycle management. Map service and level as keyword; disable dynamic mapping. Use bulk indexing and a 30 s refresh interval, because seconds of delay do not matter for logs.

A strong answer mentions: time-based indexes or data streams · shard size in the 10–50 GB range · deleting whole indexes for retention · hot/warm/cold tiers or snapshots · keyword vs text mapping · a longer refresh interval for ingestion speed.

Your turn: spot the bugs

A team makes Elasticsearch the only store for user profiles "because it can query anything". The app writes to it directly and immediately searches by email to check the user exists. Deep links let users jump to page 2,000 of results with from. The index was created with dynamic mapping and 1 shard, and is now 400 GB. What is wrong?

Compare with our answer

A strong answer mentions:

  • No system of record: there are no transactions, no unique constraint on email, and no source to reindex from. Keep profiles in a database and index a copy.
  • Searching right after writing misses the document until the next refresh (~1 s), so duplicate sign-ups slip through; uniqueness belongs in a database constraint.
  • Page 2,000 at from beyond 10,000 results fails by default, and deep from is expensive on every shard; use search_after with a point in time.
  • Dynamic mapping guessed field types (an ID mapped as text, dates as strings) and lets fields multiply; define an explicit mapping.
  • One 400 GB shard is far above 10–50 GB: slow recovery and no parallelism; reindex into ~8–16 primaries (or _split) behind an alias.
Elasticsearch on one page
  • Distributed search engine on Lucene: JSON documents → inverted index (term → postings list).
  • Mapping per field: text (analysed, searched) vs keyword (exact, filtered, aggregated). Avoid dynamic mapping in production.
  • Analyzer: tokenize → lowercase → stem/synonyms, the same at index and query time.
  • Index = primary shards (fixed at creation, aim for 10–50 GB each) + replicas (changeable; add redundancy and query capacity).
  • Near real time: translog for durability, refresh every 1 s for visibility, segments merged in the background; update = delete + reindex.
  • BM25: saturating term frequency, IDF, length normalisation; add boosts and business signals.
  • Query then fetch; deep paging with from is capped at 10,000 by default: use search_after. terms aggs are approximate across shards.
  • Sync from the database via CDC/outbox + Kafka keyed by ID, external versions, alias swaps for reindexing.
  • Not a system of record: no transactions or uniqueness; always able to rebuild from the source.
Check what stuck

1. A user edits a product title and immediately searches for the new title, but the product does not appear for about a second. Why?

2. A term query for "Nike" on the brand field returns nothing, although many products have brand "Nike". What is the most likely cause?

3. An index was created with 2 primary shards and has grown to 600 GB. What is the right way to get more primary shards?

4. An indexer consumes database changes from Kafka with several workers. Sometimes the index holds an older price than the database. What fixes it most directly?

5. A product search lets users open page 1,500 (20 results per page). Requests start failing. Why?

6. The brand facet shows "Acme: 1,204" but an exact count in the database says 1,231. The index has 8 shards. What is the likely explanation?

7. Why should Elasticsearch normally hold a copy of the data rather than the only copy?

Score 60% or more and the lesson is marked complete.

Passing the quiz also marks it complete. Progress is saved in this browser only.