Elasticsearch
How Elasticsearch works: inverted index and analyzers, shards and replicas, near-real-time refresh, BM25, aggregations, syncing from your database, and its limits.
On this page
- The problem: search that a database cannot do well
- Documents, mappings and analyzers
- Shards, replicas and the cluster
- Near real time: refresh, translog and segments
- Relevance: BM25
- Queries at scale: paging and aggregations
- Keeping it in sync with the database
- Performance numbers
- When to pick Elasticsearch, and when not
- In the interview
- Quiz
The problem: search that a database cannot do well
A shop has 50 million products. Users type "red runing shoes for men" and expect relevant results in under 200 ms, with typos forgiven, the best matches first, and counts by brand and price range down the side. In PostgreSQL, WHERE title ILIKE '%red%' AND title ILIKE '%shoes%' scans every row, cannot rank, cannot fix "runing", and computing the brand counts is another full pass.
Elasticsearch is a distributed search and analytics engine built on the Apache Lucene library. It stores JSON documents in an inverted index, which makes "documents containing these words" fast; ranks matches by relevance; computes counts and statistics (aggregations) over the matches; and spreads all of it over a cluster of machines.
The search lesson introduces inverted indexes and ranking. This lesson is about how Elasticsearch implements them, what it guarantees, and why it should almost never be your only copy of the data.
Documents, mappings and analyzers
An index (roughly, a table) holds JSON documents, each with an _id. A mapping (the schema) gives each field a type, and the type decides how it is indexed:
| Field type | Indexed as | Use for |
|---|---|---|
text | Analysed into terms (words) in an inverted index | Full-text search: titles, descriptions |
keyword | One exact term, plus columnar doc values | Filters, sorting and aggregations: brand, status, tags |
long, double, date | Numeric trees (BKD) plus doc values | Ranges, sorting, histograms |
geo_point, dense_vector | Geo and vector (HNSW graph) structures | "Within 5 km", semantic or kNN search |
A field is often mapped both ways: title as text for searching and title.raw as keyword for sorting.
An analyzer turns text into terms, at index time and again at query time, so both sides agree:
"Running Shoes, RED!"
→ tokenizer: [Running] [Shoes] [RED]
→ lowercase: [running] [shoes] [red]
→ stemmer: [run] [shoe] [red]
The inverted index then maps each term to the list of documents containing it (its postings list), with positions and frequencies. A query for "red shoes" is analysed the same way and becomes "documents in the postings lists of red and shoe". Typos are handled by fuzzy queries (terms within an edit distance of 1–2), and synonyms by a synonym filter in the analyzer.
Shards, replicas and the cluster
An index is split into primary shards, each a complete Lucene index holding a slice of the documents (chosen by hashing the routing value, by default the _id). Each primary has zero or more replica shards on other nodes.
Shards split the data; replicas copy it; queries scatter and gather
- Writes go to the primary shard, which forwards them to its replicas in parallel and acknowledges once the in-sync copies have them.
- Searches go to one copy of every shard (primary or replica), so replicas add query capacity as well as redundancy.
- The number of primary shards is fixed when the index is created (the default is 1). Changing it means
_split,_shrinkor a full reindex. The replica count can change at any time.
A cluster elects one master node (a dedicated set of three master-eligible nodes is the norm) that manages the cluster state: mappings, which shard lives where, and node membership. Since 7.0 master election uses a quorum-based protocol, which removed the old split-brain configuration trap.
Aim for shards of roughly 10–50 GB. Too few, huge shards are slow to move and recover; too many tiny shards waste memory and make every search fan out to more pieces.
Near real time: refresh, translog and segments
Lucene's segments are immutable, so new documents are not searchable the instant they are written.
Durable in the translog at once; searchable after the next refresh
The default refresh_interval is 1 second. That is the "near real time" in the documentation: a user who saves a product and immediately searches may not see it. Options when that matters: show the saved item from the response rather than re-searching; index with ?refresh=wait_for (the request returns after the next refresh, without forcing one); or, for bulk loads, set the interval to 30 s or -1 during the load to make indexing much faster.
Segments are merged in the background; flush commits them
An update is a delete plus a new document: the old version is marked deleted and disappears at the next merge. Heavy update traffic means heavy merging, so a field that changes every second (a view count, live stock) is a poor fit for the main document.
Search-idle shards and the translog setting
If a shard has had no search for 30 seconds (index.search.idle.after), scheduled refreshes are skipped until a search arrives; that first search then waits for a refresh. By default, the translog is fsynced at the end of every index or bulk request (index.translog.durability: request), so an acknowledged write survives a node crash. Setting it to async syncs every 5 seconds instead: faster, but a crash can lose acknowledged writes.
Relevance: BM25
By default, Elasticsearch ranks matches with BM25, a refinement of TF-IDF. For each query term in a document:
- Term frequency (TF): more occurrences score higher, but with diminishing returns (the parameter
k1, 1.2 by default, controls how quickly it saturates). The tenth "shoe" adds little. - Inverse document frequency (IDF): rare terms count more. "Vegan" in a shoe catalogue is more telling than "shoe".
- Length normalisation: a match in a short title beats one in a 2,000-word description (
b, 0.75 by default).
The document's score is the sum over query terms. In practice you combine it with business signals: field boosts (title^3), function_score for popularity or recency, or a learning-to-rank model on the top few hundred results. Note that IDF is computed per shard, so with tiny indexes and several shards, scores can vary slightly by shard.
Queries at scale: paging and aggregations
Query then fetch, and why deep pages are expensive
Queries mix query context (how well does it match: scored) and filter context (does it match at all: yes or no, cached, cheap). Put exact conditions like brand = nike and price < 100 in filters.
Aggregations compute summaries over the matching documents, using doc values (a column-oriented copy of each field):
{
"query": { "bool": {
"must": [{ "match": { "title": "red running shoes" } }],
"filter": [{ "range": { "price": { "lte": 100 } } }] } },
"aggs": {
"brands": { "terms": { "field": "brand", "size": 10 } },
"prices": { "histogram": { "field": "price", "interval": 25 } }
}
}
Two accuracy limits worth knowing: a terms aggregation across shards is approximate (each shard returns its own top terms, so a term that is tenth everywhere can be missed; raising shard_size reduces the error), and cardinality (distinct count) uses HyperLogLog++, accurate to within a few per cent at large counts.
Keeping it in sync with the database
Elasticsearch is usually a secondary index: the database is the system of record, and the search index is a copy shaped for queries.
Change data capture feeds the indexer; the database stays the source of truth
Ways to keep the copy current:
| Approach | How | Catch |
|---|---|---|
| Dual write from the app | Write to the database, then to Elasticsearch | A crash between the two leaves them different forever; no retry log |
| Outbox or CDC (recommended) | Read the database's change log (Debezium on the PostgreSQL WAL, MySQL binlog) into Kafka; an indexer applies it | More moving parts; seconds of lag |
| Periodic batch | Reindex rows changed since the last run | Minutes of staleness; deletes are easy to miss |
Why it is not a system of record
- No multi-document transactions, and no foreign keys or unique constraints except on
_id. - Mapping changes to existing fields need a reindex, which needs the source data.
- Writes become visible only after refresh, and replicas can briefly disagree.
- Its consistency and data-loss record has improved greatly (sequence numbers and the new cluster coordination since 6.x and 7.x), but its design goal is search, not being the last copy of money or orders.
Performance numbers
| Measure | Typical value |
|---|---|
| Search latency | ~10–100 ms for typical queries; aggregations over many documents take longer |
| Indexing per data node | roughly 10,000–50,000 small documents/s with bulk requests |
| Shard size | 10–50 GB |
| Memory | JVM heap up to ~30 GB and about half of RAM; the rest left to the OS file cache, which Lucene relies on |
| Visibility delay | ~1 s (refresh interval) |
Worked example: the shop. 50 M products at ~2 KB of indexed data each ≈ 100 GB. With 4 primary shards of ~25 GB and 1 replica, the cluster holds 200 GB. 2,000 searches/s at peak: every search touches all 4 shards, so each shard sees all 2,000. If one shard copy serves ~700–1,000 of these queries a second, each shard needs 3 copies: 1 primary and 2 replicas, 12 shard copies on 6 nodes. Measure with real queries before trusting that per-copy figure. Indexing is 500 updates/s, which is trivial. Prices and stock that change every few seconds are read from the database or a cache for the final page, not trusted from the index.
When to pick Elasticsearch, and when not
| Elasticsearch / OpenSearch | PostgreSQL full-text | Typesense / Meilisearch | Managed (Algolia) | |
|---|---|---|---|---|
| Good at | Relevance, facets, huge indexes, logs and analytics, vectors | Simple search next to your data, transactional consistency | Easy typo-tolerant site search | Instant search with no operations |
| Weak at | Operational load, being the source of truth | Ranking quality, facets at scale | Very large or complex deployments | Cost at volume, less control |
| Use when | Search or log analytics is a core feature | Tens of millions of rows and basic needs | Small to medium catalogues | Budget over control |
OpenSearch is the fork AWS started in 2021 after Elastic changed its licence; Elastic added the AGPL as an option in 2024. Both are fine interview answers.
When not to use Elasticsearch: as the primary database; for exact lookups by ID (a database or cache is simpler); for data that changes many times a second per document; for strongly consistent reads ("did my payment go through?"); and when PostgreSQL full-text search (tsvector with a GIN index) already meets your needs.
In the interview
A platform produces 2 TB of application logs a day. Engineers search the last 7 days by service, level and free text, and keep 30 days for audits. Design the indexes, shard sizing and retention, and say what you would do with logs older than 7 days.
Compare with our answer
Use time-based indexes (one per day, or data streams with rollover at ~50 GB per primary shard), so retention is "delete yesterday's index", never deleting documents. 2 TB/day of raw logs is often somewhat less indexed (compression, mapping only searchable fields as text, the rest as keyword or not indexed); assume ~2 TB/day ≈ 40 primary shards of 50 GB per day. Keep 7 days on fast "hot" nodes with 1 replica (~28 TB), then move to cheaper "warm/cold" nodes or searchable snapshots in object storage for days 8–30, managed by index lifecycle management. Map service and level as keyword; disable dynamic mapping. Use bulk indexing and a 30 s refresh interval, because seconds of delay do not matter for logs.
A strong answer mentions: time-based indexes or data streams · shard size in the 10–50 GB range · deleting whole indexes for retention · hot/warm/cold tiers or snapshots · keyword vs text mapping · a longer refresh interval for ingestion speed.
A team makes Elasticsearch the only store for user profiles "because it can query anything". The app writes to it directly and immediately searches by email to check the user exists. Deep links let users jump to page 2,000 of results with from. The index was created with dynamic mapping and 1 shard, and is now 400 GB. What is wrong?
Compare with our answer
A strong answer mentions:
- No system of record: there are no transactions, no unique constraint on email, and no source to reindex from. Keep profiles in a database and index a copy.
- Searching right after writing misses the document until the next refresh (~1 s), so duplicate sign-ups slip through; uniqueness belongs in a database constraint.
- Page 2,000 at
frombeyond 10,000 results fails by default, and deepfromis expensive on every shard; usesearch_afterwith a point in time. - Dynamic mapping guessed field types (an ID mapped as text, dates as strings) and lets fields multiply; define an explicit mapping.
- One 400 GB shard is far above 10–50 GB: slow recovery and no parallelism; reindex into ~8–16 primaries (or
_split) behind an alias.
- Distributed search engine on Lucene: JSON documents → inverted index (term → postings list).
- Mapping per field:
text(analysed, searched) vskeyword(exact, filtered, aggregated). Avoid dynamic mapping in production. - Analyzer: tokenize → lowercase → stem/synonyms, the same at index and query time.
- Index = primary shards (fixed at creation, aim for 10–50 GB each) + replicas (changeable; add redundancy and query capacity).
- Near real time: translog for durability, refresh every 1 s for visibility, segments merged in the background; update = delete + reindex.
- BM25: saturating term frequency, IDF, length normalisation; add boosts and business signals.
- Query then fetch; deep paging with
fromis capped at 10,000 by default: usesearch_after.termsaggs are approximate across shards. - Sync from the database via CDC/outbox + Kafka keyed by ID, external versions, alias swaps for reindexing.
- Not a system of record: no transactions or uniqueness; always able to rebuild from the source.
1. A user edits a product title and immediately searches for the new title, but the product does not appear for about a second. Why?
Indexed documents sit in a buffer until a refresh (1 s by default) writes a searchable segment. The translog is about durability, not visibility.
2. A term query for "Nike" on the brand field returns nothing, although many products have brand "Nike". What is the most likely cause?
term queries are not analysed, while a text field is, so the stored term differs. Map brand as keyword for exact filters and facets.
3. An index was created with 2 primary shards and has grown to 600 GB. What is the right way to get more primary shards?
The primary count is fixed at creation. Replicas copy shards rather than splitting data, and new nodes only receive existing shards.
4. An indexer consumes database changes from Kafka with several workers. Sometimes the index holds an older price than the database. What fixes it most directly?
Out-of-order application is the problem. Keying keeps one product's changes on one worker, and external versions reject older writes. Refresh and replicas do not change which write won.
5. A product search lets users open page 1,500 (20 results per page). Requests start failing. Why?
Each shard must collect from + size candidates, so cost grows with depth, and index.max_result_window defaults to 10,000. search_after pages efficiently.
6. The brand facet shows "Acme: 1,204" but an exact count in the database says 1,231. The index has 8 shards. What is the likely explanation?
Each shard reports its top terms, and terms just outside a shard's list are undercounted. Raising shard_size reduces the error. Index lag could also explain small differences, but not the shard-level approximation.
7. Why should Elasticsearch normally hold a copy of the data rather than the only copy?
Without transactions, constraints and an independent source, you cannot enforce correctness or rebuild after a mapping change. Replica writes are acknowledged by the in-sync copies; that is not the reason.
Score 60% or more and the lesson is marked complete.