What is Elasticsearch, and what are its main use cases?
Elasticsearch is a distributed search and analytics engine based on Apache Lucene. It stores JSON documents in indices and provides full-text search, structured filtering, aggregation, and vector similarity search through REST APIs and language clients; Kibana supplies a visualization and management UI.
Read full answerHide full answer
Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x
Main Use Cases
-
Application, product, and catalog search: Provide autocomplete, faceting, relevance ranking, and typo tolerance.
-
Log and observability search: Centralize logs and other operational events for investigation.
-
Security analytics: Correlate events for SIEM-style detection and investigation.
-
Operational and business analytics: Aggregate data for dashboards, reports, and trends.
-
Content, geospatial, and AI retrieval: Search documents, locations, and vector embeddings for recommendations or retrieval-augmented generation.
Architecture and Storage
-
Cluster: Elasticsearch nodes work together as a cluster. For production high availability, use at least three master-eligible nodes across failure domains.
-
Node: Nodes have roles such as
master,data,ingest,ml, and data-tier roles such asdata_hot. Every node can coordinate requests;coordinatingis not a separate role. -
Index and shard: An index is a logical collection of JSON documents divided into primary shards. Each shard is a Lucene index. Replicas provide availability and can increase search capacity.
-
Mapping and source: Mappings define field types and indexing behavior. The
_sourcefield retains the original JSON. -
Splitting: Elasticsearch and OpenSearch support splitting a primary shard into additional primaries, but the operation has prerequisites and increases resource use, so it requires capacity planning.
Indexing, Queries, and Relevance
Elasticsearch builds an inverted index that maps analyzed terms to documents. An analyzer may normalize characters, tokenize text, and normalize tokens, commonly by lowercasing them. Indexed terms are stored in immutable Lucene segments and later merged to reclaim deleted-document space and improve query performance.
Production indices should normally use explicit mappings: text for analyzed full-text fields, keyword for exact values and aggregations, and appropriate date, numeric, or boolean types elsewhere. Dynamic mapping can create unexpected field types or mapping explosions in unpredictable datasets.
| Query | Purpose |
|---|---|
match |
Full-text search using an analyzer |
term |
Exact match without analysis |
bool |
Combine must, should, filter, and must_not clauses |
range |
Numeric, date, or IP boundaries |
exists |
Find documents containing a field |
wildcard or regexp |
Pattern matching; potentially expensive on large fields |
Clauses in a bool filter context are eligible for the query cache and do not contribute to relevance scores. By default, match and phrase queries use BM25 scoring, which considers term frequency, inverse document frequency, and field-length normalization.
Analytics, Ingestion, Lifecycle, and Vectors
-
Aggregations: Compute counts, averages, percentiles, and grouped summaries. Use
termsfor grouping anddate_histogramfor time buckets; high-cardinality aggregations can consume significant memory. -
Ingestion: Index documents directly or use ingest pipelines for parsing and enrichment. Logstash, Elastic Agent, and Kafka-compatible connectors are common pipeline components.
-
Lifecycle management: Elasticsearch ILM automates actions such as rollover, shrink, force merge, and deletion. OpenSearch provides similar functionality through Index State Management, or ISM.
-
Vector search: Elasticsearch 8.x supports
dense_vector, approximate kNN queries, and exactscript_scorecalculations. OpenSearch usesknn_vectorand its k-NN plugin.
Production Operations and Security
-
Shard sizing: Aim for primary shards commonly in the 10–50 GB range for typical search workloads, but benchmark against document size, query mix, and retention. Avoid many tiny shards and monitor shard count per node.
-
Distribution and routing: Spread primaries across nodes. Replicas increase read capacity, while custom routing can create hotspots and should be used only with deliberate key design.
-
Cluster health:
greenmeans all primaries and replicas are allocated;yellowmeans primaries are available but replicas are not;redmeans at least one primary is unavailable. -
Capacity: Long JVM garbage-collection pauses, excessive heap use, and disk watermarks can indicate shard-count, aggregation, or capacity problems. At the flood-stage watermark, Elasticsearch can make indices read-only.
-
Monitoring: Track cluster health, unassigned shards, indexing and search latency, rejection counts, JVM GC, disk usage, and snapshot failures.
-
Backups: Configure snapshot repositories and test restores. Snapshots are incremental and cluster-aware, but they are not real-time copies of the original cluster.
-
Upgrades: Perform rolling upgrades one node at a time, follow the product’s compatibility requirements, and verify health and behavior after each stage.
-
Security: Elasticsearch 8.x enables TLS and role-based access control by default. OpenSearch provides comparable controls through its Security plugin. In either system, restrict privileges, avoid public exposure, and place Elasticsearch behind an appropriately secured gateway or load balancer where necessary.