Skip to content
Cloud, DevOps & Infrastructure

Elasticsearch interview questions and answers

78 questions, 30 with code. About 6 h 47 min to read them all.

By TestLab · Updated

Plan for an interview
78 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Elasticsearch, and what are its main use cases?

Elasticsearch is a distributed search and analytics engine based on Apache Lucene. It stores JSON documents in indices and provides full-text search, structured filtering, aggregation, and vector similarity search through REST APIs and language clients; Kibana supplies a visualization and management UI.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x

Main Use Cases

  • Application, product, and catalog search: Provide autocomplete, faceting, relevance ranking, and typo tolerance.

  • Log and observability search: Centralize logs and other operational events for investigation.

  • Security analytics: Correlate events for SIEM-style detection and investigation.

  • Operational and business analytics: Aggregate data for dashboards, reports, and trends.

  • Content, geospatial, and AI retrieval: Search documents, locations, and vector embeddings for recommendations or retrieval-augmented generation.

Architecture and Storage

  • Cluster: Elasticsearch nodes work together as a cluster. For production high availability, use at least three master-eligible nodes across failure domains.

  • Node: Nodes have roles such as master, data, ingest, ml, and data-tier roles such as data_hot. Every node can coordinate requests; coordinating is not a separate role.

  • Index and shard: An index is a logical collection of JSON documents divided into primary shards. Each shard is a Lucene index. Replicas provide availability and can increase search capacity.

  • Mapping and source: Mappings define field types and indexing behavior. The _source field retains the original JSON.

  • Splitting: Elasticsearch and OpenSearch support splitting a primary shard into additional primaries, but the operation has prerequisites and increases resource use, so it requires capacity planning.

Indexing, Queries, and Relevance

Elasticsearch builds an inverted index that maps analyzed terms to documents. An analyzer may normalize characters, tokenize text, and normalize tokens, commonly by lowercasing them. Indexed terms are stored in immutable Lucene segments and later merged to reclaim deleted-document space and improve query performance.

Production indices should normally use explicit mappings: text for analyzed full-text fields, keyword for exact values and aggregations, and appropriate date, numeric, or boolean types elsewhere. Dynamic mapping can create unexpected field types or mapping explosions in unpredictable datasets.

Query Purpose
match Full-text search using an analyzer
term Exact match without analysis
bool Combine must, should, filter, and must_not clauses
range Numeric, date, or IP boundaries
exists Find documents containing a field
wildcard or regexp Pattern matching; potentially expensive on large fields

Clauses in a bool filter context are eligible for the query cache and do not contribute to relevance scores. By default, match and phrase queries use BM25 scoring, which considers term frequency, inverse document frequency, and field-length normalization.

Analytics, Ingestion, Lifecycle, and Vectors

  • Aggregations: Compute counts, averages, percentiles, and grouped summaries. Use terms for grouping and date_histogram for time buckets; high-cardinality aggregations can consume significant memory.

  • Ingestion: Index documents directly or use ingest pipelines for parsing and enrichment. Logstash, Elastic Agent, and Kafka-compatible connectors are common pipeline components.

  • Lifecycle management: Elasticsearch ILM automates actions such as rollover, shrink, force merge, and deletion. OpenSearch provides similar functionality through Index State Management, or ISM.

  • Vector search: Elasticsearch 8.x supports dense_vector, approximate kNN queries, and exact script_score calculations. OpenSearch uses knn_vector and its k-NN plugin.

Production Operations and Security

  • Shard sizing: Aim for primary shards commonly in the 10–50 GB range for typical search workloads, but benchmark against document size, query mix, and retention. Avoid many tiny shards and monitor shard count per node.

  • Distribution and routing: Spread primaries across nodes. Replicas increase read capacity, while custom routing can create hotspots and should be used only with deliberate key design.

  • Cluster health: green means all primaries and replicas are allocated; yellow means primaries are available but replicas are not; red means at least one primary is unavailable.

  • Capacity: Long JVM garbage-collection pauses, excessive heap use, and disk watermarks can indicate shard-count, aggregation, or capacity problems. At the flood-stage watermark, Elasticsearch can make indices read-only.

  • Monitoring: Track cluster health, unassigned shards, indexing and search latency, rejection counts, JVM GC, disk usage, and snapshot failures.

  • Backups: Configure snapshot repositories and test restores. Snapshots are incremental and cluster-aware, but they are not real-time copies of the original cluster.

  • Upgrades: Perform rolling upgrades one node at a time, follow the product’s compatibility requirements, and verify health and behavior after each stage.

  • Security: Elasticsearch 8.x enables TLS and role-based access control by default. OpenSearch provides comparable controls through its Security plugin. In either system, restrict privileges, avoid public exposure, and place Elasticsearch behind an appropriately secured gateway or load balancer where necessary.

2

How does Elasticsearch differ from OpenSearch?

Elasticsearch and OpenSearch are distributed search and analytics engines built on Apache Lucene. They share much of their architecture and API, but differ in governance, licensing, lifecycle tooling, ecosystem integration, and the availability of newer capabilities.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Differences

Area Elasticsearch OpenSearch
Origin Developed by Elastic Forked from Elasticsearch 7.10.2 and released by AWS in 2021
Governance Primarily controlled by Elastic Community project governed through the Linux Foundation
License Source-available: earlier 8.x releases use Elastic License 2.0 and SSPL terms, while 8.16+ also offers AGPLv3 Apache License 2.0
Ecosystem Elastic Stack, including Kibana, Logstash, Beats, Elastic Agent, and Elastic Cloud OpenSearch Dashboards, OpenSearch plugins, OpenSearch Agent, and AWS OpenSearch Service
Lifecycle Index Lifecycle Management (ILM) Index State Management (ISM)
Vector search dense_vector fields with Lucene HNSW support knn_vector fields through the OpenSearch k-NN plugin and supported vector engines
Extensions Elastic-specific plugins and subscription features OpenSearch plugins, including security, anomaly detection, alerting, and k-NN

OpenSearch began as an Elasticsearch fork, but the projects now evolve independently. Elasticsearch features are not automatically available in OpenSearch, and OpenSearch-specific behavior must not be assumed to work in Elasticsearch.

Shared Architecture and Search Features

Both engines use a similar architecture:

  • A cluster contains multiple nodes for availability and fault tolerance.

  • An index is divided into primary shards and optional replica shards.

  • Each shard is a Lucene index containing an inverted index and immutable segments. Background merges combine segments to improve query performance and reclaim space.

  • Replicas provide redundancy. A cluster is green when all primaries and replicas are allocated, yellow when replicas are unassigned, and red when at least one primary is unavailable.

  • Both provide JSON mappings, analyzed text fields, exact-match keyword fields, custom analyzers, data streams, ingest pipelines, aggregations, and BM25 relevance scoring.

Their common Query DSL includes:

  • match for analyzed text searches.

  • term for exact values, normally against a keyword field.

  • bool for combining must, should, and filter clauses.

  • filter queries, which run in filter context, do not affect the relevance score, and are cacheable when their usage pattern permits caching.

Both also provide approximate nearest-neighbor vector search, but their field types, plugin behavior, filtering semantics, and tuning options differ.

Operations, Monitoring, and Upgrades

Elasticsearch ILM manages rollover, deletion, retention, and data-tier allocation for hot-warm-cold architectures. OpenSearch ISM supports similar lifecycle actions through scheduled policies, but its configuration model and available actions are not identical.

For monitoring, Elasticsearch commonly uses Elastic Stack Monitoring, metrics, logs, and Elastic Agent. OpenSearch uses its monitoring stack, OpenSearch Dashboards, and OpenSearch Agent. Both should be monitored with Prometheus-compatible metrics, JVM metrics, disk watermarks, shard allocation, garbage collection, search latency, indexing throughput, and rejected requests.

Both support snapshots to repositories such as S3. Snapshot restoration should be tested rather than merely checking that snapshots exist. A snapshot created by one distribution must not be assumed to be restorable into the other; use the documented reindex or export/import migration path.

Production upgrades normally use each product’s documented rolling procedure. However, some supported version transitions permit a full-cluster restart, so the source and target compatibility matrices must be checked. In a rolling upgrade, upgrade replica-bearing nodes before primaries and never mix Elasticsearch and OpenSearch nodes in one cluster.

Security and Governance

Both products support TLS, authentication, role-based access control, and audit logging, although availability can depend on the subscription or security-plugin configuration. Production deployments should:

  • Enable TLS for node transport and client traffic.

  • Use least-privilege RBAC roles.

  • Restrict endpoints to private networks or a controlled gateway.

  • Store credentials in a secrets manager rather than configuration files.

  • Enable audit logging where required and supported.

  • Encrypt data at rest, volumes, and snapshot repositories using the appropriate product or infrastructure features.

  • Replace demonstration certificates and default credentials before production use.

OpenSearch’s Apache 2.0 license makes embedding and redistribution easier than under Elastic’s source-available terms. Managed-service terms, infrastructure costs, and support contracts remain separate considerations.

Production Best Practices

  • Size shards deliberately: Roughly 10–50 GB per primary shard is a common starting point for read-heavy data; high-churn workloads may require smaller shards.

  • Use replicas for availability: One replica is common, but replicas consume storage and indexing resources.

  • Avoid oversharding: Excessive shard counts increase cluster-state, heap, and scheduling overhead.

  • Distribute primaries across failure domains: Use node attributes and allocation awareness so a node or rack failure does not remove every primary copy of an index.

  • Plan capacity and retention: Monitor watermarks, rejected requests, merge pressure, and query latency; use ILM or ISM to implement rollover and deletion policies.

  • Avoid blind API portability: Test mappings, ingest processors, lifecycle policies, plugins, snapshots, and upgrade procedures against the exact target version.

3

How does Elasticsearch build on Apache Lucene?

Apache Lucene is the embedded search library underlying Elasticsearch: every shard is backed by a Lucene index, while Elasticsearch adds distributed indexing, cluster coordination, replication, mappings, aggregations, and operational tooling.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Relationship

Lucene is a Java library designed to run inside an application. It does not provide a distributed control plane, shard replication, or automatic failover.

Each Elasticsearch/OpenSearch data node hosts one or more shards:

  • Index: A logical collection of documents, such as orders.

  • Primary shard: One partition of an index, implemented as a Lucene index.

  • Replica shard: A copy of a primary shard allocated to another node.

  • Node: An Elasticsearch/OpenSearch process that can hold multiple shards and coordinate operations.

An index with three primaries and one replica therefore creates three primary Lucene indexes and three replica copies. Shards are distributed across nodes; replicas provide redundancy and additional search capacity.

How Lucene Stores and Searches Data

Lucene stores searchable fields in an inverted index, mapping analyzed terms to matching documents. Doc values provide column-oriented values for sorting, aggregations, and some filtering operations. Lucene stored fields are available only where store: true; _source is separately maintained compressed JSON and is not a Lucene stored field.

Lucene indexes consist of immutable segments:

  1. New writes enter an in-memory indexing buffer.

  2. A refresh opens a new searcher, making buffered operations searchable; it may also flush a new Lucene segment.

  3. Lucene merges segments locally to reduce search overhead and reclaim space occupied by deleted documents.

  4. Elasticsearch/OpenSearch coordinate refresh visibility, replication, and recovery across the cluster.

Refresh controls search visibility, not durability; the translog provides recovery for operations that have not yet been committed into Lucene segments.

A match query analyzes query text using the field’s analyzer, while a term query matches an exact term without analysis. bool combines query clauses, and clauses in bool.filter execute in filter context without contributing to the score. Elasticsearch’s default relevance algorithm, BM25, uses Lucene’s similarity framework.

Features Elasticsearch Adds

Elasticsearch adds distributed-system behavior around Lucene:

  • Cluster coordination: Maintains cluster state and elects a master-eligible node.

  • Shard allocation: Assigns primaries and replicas to nodes according to allocation rules and failure-domain awareness.

  • Distributed search: Sends requests to relevant shards and merges their local results.

  • Aggregations: Computes metrics and buckets across shards.

  • Mappings and analyzers: Define field types, indexing behavior, and text processing.

  • Ingestion: Supports bulk indexing, data streams, and ingest pipelines.

  • Lifecycle management: Uses index lifecycle management policies and rollover aliases.

  • Vector search: dense_vector fields support exact searches and approximate k-nearest-neighbor retrieval backed by Lucene HNSW indexes.

Distributed Failure Behavior

Replication normally places replicas on different nodes, but it does not automatically place them in different racks or availability zones.

  • Green: Every primary and replica is allocated.

  • Yellow: Every primary is allocated, but at least one replica is missing.

  • Red: At least one primary is unavailable, making part of the index unserviceable.

Configure shard allocation awareness or anti-affinity when replicas must survive an entire rack or zone failure. Elasticsearch/OpenSearch apply disk watermarks to avoid disk exhaustion. At the flood-stage threshold, writes can be blocked and the affected index marked read-only. Excessive merges, field-data growth, large aggregations, and deep pagination can also exhaust disk space, heap, CPU, or I/O capacity.

OpenSearch Differences

OpenSearch was forked from Elasticsearch 7.10.2 and initially used Lucene 9. OpenSearch 2.x and 3.x retain the same shard-based Lucene architecture, inverted indexes, analyzers, BM25 relevance, and aggregations.

The distributions differ in plugins, APIs, security configuration, and feature development. OpenSearch commonly uses Index State Management policies rather than Elasticsearch ILM. Its vector-search plugin exposes engine-specific settings and tuning options, so Elasticsearch vector APIs and index settings should not be assumed to work unchanged in OpenSearch.

Production Operations

  • Monitoring: Track health, unassigned shards, allocation explanations, disk watermarks, JVM heap, CPU, merge pressure, indexing and search latency, rejected requests, and field-data usage. Green means all shards are allocated, not that the cluster has spare capacity; yellow should reflect an explicitly accepted risk, while red requires immediate action.

  • Index management: Use index templates to enforce consistent mappings and settings. Use data streams with rollover and lifecycle policies to control index proliferation and retention.

  • Recovery: Take repository-backed snapshots and test restores. Snapshots provide recovery points independent of replica placement, but replicas remain necessary for local redundancy and rapid failover.

  • Shard sizing: Benchmark with representative documents, cardinality, queries, and retention. Avoid both many tiny shards and oversized shards; a shard in the tens-of-gigabytes range is a common starting point for conventional search workloads, not a universal limit. Primary shard count is fixed at index creation, so over-sharding is difficult to undo and normally requires reindexing or splitting.

  • Security and upgrades: Use TLS, role-based access control, least-privilege API keys, and appropriate audit logging. Before an upgrade, take a recoverable snapshot, follow the supported upgrade path, account for plugin compatibility, and monitor recovery and reindexing pressure.

4

What are ELK and the Elastic Stack?

ELK is the traditional combination of Elasticsearch, Logstash, and Kibana. The broader Elastic Stack adds components such as Beats and Elastic Agent for collecting, processing, searching, visualizing, and securing operational data.

Read full answer

Applies toElasticsearch 8.x, Kibana 8.x, Logstash 8.x, Elastic Agent 8.x, and OpenSearch 2.x

Core Components

Component Purpose
Elasticsearch Distributed search and analytics engine built on Apache Lucene
Kibana Web interface for dashboards, Dev Tools, indexing, and management
Logstash Server-side data processing pipeline for parsing, enrichment, and transformation
Beats Lightweight data shippers such as Filebeat, Metricbeat, and Auditbeat
Elastic Agent Unified agent that collects logs, metrics, and other telemetry and can integrate with Fleet

“ELK” usually means using all three original products, but modern deployments often omit Logstash. Elastic Agent or Beats can ingest data directly into Elasticsearch, while Logstash remains useful for complex transformations and connecting to less conventional sources.

Architecture and Data Flow

An Elasticsearch cluster contains multiple nodes. Nodes can perform several roles:

  • Master-eligible nodes manage cluster metadata and coordinate shard allocation.

  • Data nodes store shards and execute searches.

  • Coordinating nodes distribute requests and aggregate results.

  • Ingest nodes run ingest pipelines before documents are indexed.

An index is a logical collection of JSON documents. It is divided into primary shards and optional replica shards. A shard is an actual Lucene index containing an inverted index, which maps terms to document frequencies. Lucene also stores the document data, and periodically merges immutable segments into larger segments for faster searching and fewer files.

During indexing, the coordinating node selects a primary shard, which forwards the operation to the replicas. The primary confirms durability through the translog. Indexing is near real time: documents become searchable after a refresh, commonly about one second by default, rather than after every individual write.

Search and Data Capabilities

  • Mappings define field types and index behavior. Text fields are analyzed, while keyword fields are stored as exact values.

  • Analyzers tokenize and normalize text. Standard analyzers suit most English content; custom analyzers handle language-specific or domain-specific processing.

  • The Query DSL supports full-text queries such as match and exact-value queries such as term. The bool query combines must, should, filter, and must_not clauses, so a typical search puts a match clause in must and an exact-value term clause in filter, and adds size for pagination.

  • Filters are cached and do not contribute to relevance scoring. Full-text match queries use BM25, Elasticsearch’s default relevance model in 8.x.

  • Aggregations calculate metrics such as averages and percentiles or summarize data by terms, dates, ranges, and other dimensions; a terms aggregation is normally run on a keyword field.

  • Ingest pipelines can parse, normalize, enrich, or route documents before indexing.

  • Index Lifecycle Management (ILM) automates rollover, deletion, snapshots, and replica management.

  • Vector search stores dense vectors in dense_vector fields and supports approximate nearest-neighbor retrieval through vector-indexing techniques such as HNSW. Exact script_score searches are also possible for smaller datasets.

Production Operations

  • Shard sizing: Avoid both tiny and enormous shards. Shards in the approximate 10–40 GB range are often a starting point, but workload, hardware, retention, and query patterns must be benchmarked. Remember that replicas also consume disk, memory, and I/O.

  • Cluster health: Green means all primaries and replicas are assigned; yellow means primaries are assigned but some replicas are not, which is normal for a single-node cluster; red means at least one primary is unavailable.

  • Capacity planning: Monitor disk headroom, JVM heap and garbage collection, indexing rates, search latency, rejected tasks, CPU, and shard allocation. Elasticsearch stops assigning shards to nodes once the 90% disk watermark is crossed, and at the 95% flood-stage watermark it puts affected indices into read-only mode.

  • Backups: Use snapshots stored in a snapshot repository. Snapshot retention and repository redundancy must be designed separately from the cluster; snapshots are not a substitute for testing restores.

  • Upgrades: Follow a tested rolling-upgrade procedure and upgrade Kibana and supporting clients appropriately. Take a snapshot, verify compatibility, upgrade one node at a time, and watch health and performance throughout the process.

  • Security: Enable TLS, RBAC, audit controls, and encryption at rest. Use least-privilege API keys for ingestion and querying rather than sharing administrator credentials.

Elastic Stack versus OpenSearch

OpenSearch originated from the Elasticsearch 7.10.2 codebase and is now maintained separately. It uses OpenSearch Dashboards instead of Kibana, provides its own plugins and security tooling, and has a different vector-search implementation. OpenSearch supports many similar concepts—clusters, shards, mappings, BM25, aggregations, ILM-style lifecycle automation, and security—but compatibility is not guaranteed. Elasticsearch 8.x features, especially newer vector and query capabilities, should not be assumed to exist unchanged in OpenSearch 2.x.

5

How do Elasticsearch clusters, nodes, indices, and documents relate?

Elasticsearch data model organizes JSON data across a distributed cluster of nodes. A cluster contains nodes, nodes hold index shards, and each index stores documents distributed among those shards; OpenSearch follows essentially the same model.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Components

Component Meaning Production role
Cluster A group of nodes coordinated through shared cluster state and a discovery mechanism Distributes indexing and search work across machines
Node An Elasticsearch or OpenSearch server process running on the JVM Stores shard data, coordinates requests, or participates in cluster management
Index A logical collection of documents with shared mappings and settings Represents an entity such as orders, events, or products
Document A JSON object containing source data plus metadata such as _id, _index, and _version The unit that is indexed, retrieved, updated, and deleted
Shard A Lucene index containing a subset of an index’s documents Provides horizontal storage and search capacity
Replica A copy of a primary shard Improves availability and can increase search throughput

By default, an index has one primary shard and one replica; both counts are configurable, and zero replicas is valid. The primary coordinates writes, while replicas apply replicated operations. If a primary fails, an in-sync replica can be promoted. Replicas improve availability and search throughput but add storage use and provide no independent redundancy when placed in the same failure domain.

Nodes can have specialized roles. Some hold data, some coordinate requests or perform ingestion, and others participate in cluster management. Small clusters commonly combine roles, while large clusters separate these workloads to reduce contention.

How Data Is Stored and Queried

Documents are routed to a primary shard, normally by hashing the document ID and configured routing value. Replicas belong to the same shard group, and the cluster reallocates shards as nodes join, leave, or fail.

Each shard is backed by Lucene segments. Writes enter the in-memory index and are periodically flushed into immutable segments; merges combine segments and reclaim deleted space.

Elasticsearch uses an inverted index to map analyzed terms to document positions:

  • A match query analyzes search text using the field’s analyzer.

  • A term query searches for an exact term without analyzing the query input.

  • A bool query combines must, should, filter, and must_not clauses.

  • Filters are cached more aggressively because they do not contribute to relevance scoring.

  • Text relevance normally uses BM25.

  • Aggregations summarize indexed values using metrics such as sum, avg, and cardinality.

Mappings define how fields are indexed. Dynamic mapping can cause mapping explosion when arbitrary field names are written. Use explicit mappings, keyword fields for exact matching, and text fields for analyzed full-text search.

Ingestion and Vector Search

  • Ingestion: Clients can send documents directly, through Beats or other agents, or through ingest pipelines. Nodes with the ingest role execute pipeline processors on each document before indexing. Heavy processors such as grok, script, or enrichment consume CPU and memory, so these nodes require adequate resources and monitoring.

  • Vector search: Elasticsearch 8.x uses a dense_vector mapping, while OpenSearch uses knn_vector. Both support approximate k-nearest-neighbor queries and exact script-based scoring. Their request schemas differ, and k, candidate-set size, memory use, recall, and latency must be benchmarked for the workload.

Failure Modes and Cluster Health

Cluster health summarizes primary and replica allocation:

  • Green: All primary and replica shards are assigned.

  • Yellow: All primaries are active, but at least one replica is not.

  • Red: At least one primary shard is unavailable.

Common causes include disk exhaustion, excessive shard counts, garbage-collection pressure, unsuitable heap sizing, and insufficient resources during rebalancing. Flood-stage disk watermarks can make indices read-only, while excessive concurrent work can cause thread-pool rejections.

Keep primary shards around 10–50 GB as a starting point, adjusted for the workload, and avoid thousands of tiny shards. Shards add heap and coordination overhead. Use aliases with rollover or reindex-and-alias workflows instead of changing shard layout in place.

Scaling, Lifecycle, and Operations

Use data streams for append-only, time-based workloads. Elasticsearch ILM handles rollover, retention, deletion, and a snapshot action; the separate Snapshot Lifecycle Management feature manages scheduled repository snapshots. OpenSearch ISM provides similar lifecycle controls, while OpenSearch snapshot management is separate.

Monitor shard allocation, disk and JVM pressure, garbage collection, search latency, indexing throughput, rejected requests, unassigned shards, merge pressure, and field-data usage. Useful endpoints include _cluster/health, _cat/shards, _nodes/stats, and _cat/indices.

Schedule incremental snapshots to an external repository such as S3. A snapshot is a point-in-time, cluster-consistent copy, but restores must be tested. Rolling upgrades can cause shard movement or temporary yellow health, so monitor recovery before advancing to the next node.

Security and Distribution Differences

Use TLS for node-to-node and client traffic, enable RBAC and audit logging, rotate API keys, and avoid exposing port 9200 publicly. Apply least privilege to ingest, read, administrative, and snapshot roles.

Elasticsearch and OpenSearch share the core cluster, shard, Lucene, Query DSL, BM25, aggregation, and vector-search foundations, but they are not drop-in equivalents. Plugins, security implementation, lifecycle tooling, and operational defaults differ, so use each distribution’s own deployment and upgrade guidance.

6

How do Elasticsearch shards and replicas work?

In Elasticsearch and OpenSearch, a shard is a Lucene index containing a portion of an index’s documents. A replica shard is a logical copy of a primary shard that provides redundancy and additional read capacity, and can be promoted if the primary fails.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Architecture and Allocation

An index is divided into one or more primary shards. Each primary shard can have a configured number of replica shards.

Shard type Role Failure behavior
Primary shard Handles writes and is the authoritative copy Can be replaced by a promoted replica
Replica shard Holds a copy and can serve searches Can be promoted if the primary fails
Unassigned shard Exists but has no allocated copy Indicates an allocation or capacity problem
  • Node placement: Each shard copy is assigned to a data node. Copies of the same shard—the primary and its replicas—cannot be allocated to the same node.

  • Lucene structure: Each shard is a Lucene index composed of immutable segments. New documents create additional segments, which background merges later consolidate. Lucene’s inverted-index structures can be memory-mapped or stored on disk; Elasticsearch also maintains in-memory structures.

  • Shard granularity: Shards are the unit of data storage, parallel search, and shard-level recovery. Replicas improve availability and distribute searches but do not divide a logical shard further.

  • Shard counts: The primary-shard count is fixed when an index is created. It can be changed through a controlled reindex or split. Replica counts can be adjusted dynamically.

An index with three primary shards and two replicas has nine shard copies. With no other allocation constraints, all nine can be allocated across three data nodes, placing each primary and its two replicas on three different nodes.

Routing, Writes, and Searches

Documents are assigned to shards using a hash of their routing value and the shard count. The default routing value is the document _id, so documents with the same routing value normally reach the same shard. Custom routing can improve locality, but a poorly distributed routing key can create a hot shard.

Write operations are routed to the target primary shard, which then replicates changes to its replicas. Searches normally query one available copy of each relevant shard, whether primary or replica. A coordinating node gathers the partial results, combines them, and applies the query’s requested sorting and aggregation logic.

Sizing and Lifecycle Planning

There is no hard maximum shard size, but 10–50 GB per primary shard is a common starting range. Validate it against indexing throughput, query latency, recovery time, available disk, and cluster size.

  • Too many small shards: Increase heap, cluster-state, and scheduling overhead and create many small Lucene segments.

  • Too few large shards: Concentrate traffic, slow recovery and relocation, and limit parallelism.

  • Replica overhead: Each replica consumes approximately the disk and resource capacity of its primary.

  • Capacity headroom: Keep sufficient free disk above the configured flood-stage watermarks for allocation and recovery.

  • Lifecycle management: Use time-based indices and rollover policies to delete old data or move it to an appropriate warm or read-only phase. Elasticsearch ILM and OpenSearch ISM provide different implementations.

Health and Failure Behavior

Cluster health summarizes shard allocation:

  • Green: Every primary and replica shard is allocated.

  • Yellow: All primaries are allocated, but at least one replica is unavailable.

  • Red: At least one primary is unavailable, so part of the index cannot be served normally.

A single-node development cluster commonly appears yellow when replicas are enabled because a replica cannot be allocated to the same node as its primary. Production clusters need enough nodes and appropriate allocation settings to distribute replicas across failure domains.

When a primary fails, an eligible replica may be promoted, and the cluster can allocate another replica to restore redundancy. This requires sufficient disk capacity and healthy cluster coordination. Replicas protect against node and zone failures, but they do not protect against corruption, accidental deletion, or invalid mappings. Use repository-based snapshots and test restores for recovery from destructive operations.

Production Practices and Differences

  • Placement: Use allocation awareness or filtering to distribute primary and replica copies across nodes and availability zones.

  • Monitoring: Alert on red health, unexpected yellow health, unassigned shards, disk and JVM pressure, long garbage collections, rejected tasks, indexing latency, and search latency.

  • Upgrades: Perform rolling upgrades one node at a time, checking health and allocation state between steps.

  • Security: Enable TLS for node transport and client traffic, apply least-privilege RBAC, restrict network access, and use encryption at rest where required.

  • Elasticsearch versus OpenSearch: Their shard and replication models are fundamentally equivalent. Elasticsearch uses ILM and its native security features; OpenSearch uses ISM and its own security-plugin ecosystem. Operator APIs and advanced features should not be assumed to be interchangeable.

7

What node roles exist in Elasticsearch and OpenSearch?

Node roles determine the cluster operations performed by an Elasticsearch or OpenSearch node. A node may have several roles, and a node with no configured roles is a coordinating-only node.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.4–3.x

Elasticsearch 8.x Node Roles

Role Responsibility
master Manages cluster state, index creation, shard allocation, and cluster membership. Only master-eligible nodes participate in elections.
data Stores and processes shards without a specialized data-tier assignment. Explicit tier roles provide better workload isolation.
data_content Stores content-tier shards used for searches, aggregations, and related operations.
data_hot Stores frequently modified and frequently searched data, typically receiving new writes.
data_warm Stores less frequently searched data while keeping it fully searchable.
data_cold Stores infrequently accessed data while retaining searchability.
data_frozen Hosts the lowest-cost searchable tier, commonly using fully mounted searchable snapshots.
ingest Runs ingest pipelines that parse, transform, or enrich documents before indexing.
ml Runs machine-learning workloads such as anomaly detection.
transform Runs batch and continuous transforms. (Data frame analytics jobs run on ml nodes.)
remote_cluster_client Connects to remote clusters for cross-cluster search and related remote operations.
remote_cluster_server Receives cross-cluster requests from remote clusters. Assign it only to nodes intended to serve those requests.
voting_only Used together with master: the node votes in master elections but cannot be elected master, which makes it a cheap tiebreaker.
No configured role Creates a coordinating-only node that routes requests, gathers shard responses, and combines results without holding shards. Nodes with other roles also coordinate by default.

OpenSearch Node Roles

OpenSearch role Responsibility
cluster_manager Manages cluster state and shard allocation. It replaces the legacy master name used by earlier OpenSearch releases.
data Stores and searches data without a specialized warm-tier assignment.
warm Hosts warm-tier indices, including data managed with remote-backed storage and searchable snapshots.
ingest Runs ingest pipelines.
ml Runs machine-learning workloads.
remote_cluster_client Connects to remote OpenSearch clusters for supported cross-cluster operations.
No configured role Creates a coordinating-only node. There is no separate coordinating-only role to configure.

OpenSearch does not expose Elasticsearch’s data_content, data_hot, data_cold, data_frozen, remote_cluster_server, voting_only, or transform roles. Its principal storage separation is between general data nodes and dedicated warm nodes.

How Roles Affect Architecture

Master-eligible nodes maintain the authoritative cluster state and decide where shards run. Data nodes hold Lucene shards, ingest nodes execute pipelines, and all nodes can coordinate client requests unless excluded by their configuration.

A common production deployment uses:

  • Three management nodes: Elasticsearch master or OpenSearch cluster_manager nodes with the required voting configuration.

  • Hot data nodes: Handle recent writes and frequent searches.

  • Warm or frozen nodes: Hold less frequently accessed data on lower-cost storage while preserving searchability.

  • Ingest nodes: Isolate pipeline parsing and enrichment overhead.

  • Coordinating nodes: Handle client traffic without allocating shard storage.

Management nodes should be distributed across independent failure domains. Dedicated coordinating nodes can isolate bulk indexing and fan-out searches, but they add network hops and can become bottlenecks.

Production and Security Guidance

  • Use explicit Elasticsearch tier roles: Assign specialized tier roles according to access patterns. Index Lifecycle Management normally moves indices between searchable tiers; manual reindexing should be exceptional.

  • Plan OpenSearch warm migration carefully: Use the OpenSearch warm role and manage lifecycle transitions with Index State Management rather than assuming Elasticsearch ILM behavior is identical.

  • Size shards pragmatically: Shards in the tens-of-gigabytes range are often reasonable, but validate against indexing rate, query fan-out, segment count, recovery time, and heap pressure. Frozen searchable snapshots have different sizing characteristics.

  • Protect management capacity: Use at least three management nodes, monitor heap and garbage collection, and keep them isolated from data workloads.

  • Apply least privilege: Enable TLS and Elasticsearch Security or the OpenSearch Security plugin. Restrict APIs with security roles and role mappings. Use node attributes for allocation and placement rules, not as the primary authorization mechanism.

  • Limit cross-cluster access: Assign remote_cluster_client only where outbound remote access is required and remote_cluster_server only where inbound remote requests are expected.

  • Monitor continuously: Track disk watermarks, heap pressure, indexing and search latency, rejected requests, unassigned shards, and cluster-manager quorum. Useful APIs include GET /_cluster/health, GET /_cat/shards?v, and GET /_nodes/stats.

Failure Modes and Trade-offs

Loss of a data node causes shard recovery when replicas are available, normally including promotion of a replica. Without an available primary or replica, the affected index remains red and reads or writes for unavailable shards can fail.

Disk allocation safeguards can block writes to affected indices, including setting a read-only block when the flood-stage watermark is reached (the high watermark only stops new shards being allocated to the node and starts relocation). This is an index-level protection mechanism rather than an automatic cluster-wide read-only state.

Loss of the required master or cluster-manager quorum prevents safe cluster-state updates, although existing shards may remain readable. Dedicated coordinating nodes improve workload isolation, but their request queues, garbage collection, network utilization, and CPU saturation require active monitoring.

8

How does master election work in Elasticsearch?

Elasticsearch elects one master-eligible node to coordinate cluster metadata, index operations, and shard-allocation decisions. A candidate becomes master only after receiving votes from a majority of nodes in the cluster’s voting configuration, which prevents two nodes from independently managing the same cluster state.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 3.x

Election Process

  1. Node discovery: Nodes find peers through discovery.seed_hosts. Discovery establishes the cluster and identifies master-eligible nodes.

  2. Election trigger: A master-eligible node starts an election when it cannot contact a valid master and determines that the existing master is unavailable.

  3. Candidacy: The candidate advances the cluster’s election term and publishes its candidacy. The term prevents stale nodes from competing with a newer election.

  4. Voting: Voting nodes compare the candidate’s term and cluster state with their own state. A candidate needs votes from a majority of the voting configuration.

  5. Leadership: After winning the vote, the candidate becomes master and publishes the authoritative cluster state. It then coordinates index creation and deletion, mappings, index templates, shard allocation, and other metadata operations.

The elected master is also the cluster coordinator. Data nodes normally do not vote and can continue serving searches and writes while the master changes.

Quorum and Fault Tolerance

The required quorum is a majority of the voting configuration, which is not necessarily the same as the total number of master-eligible nodes.

Master-eligible nodes Votes required Failures tolerated
1 1 0
2 2 0
3 2 1
4 3 1
5 3 2

A two-node setup provides no additional availability because losing either node loses quorum. Three or five dedicated master-eligible nodes are common choices. Replica quorum and shard allocation are separate concerns from master-election quorum.

Bootstrap Differences

  • Elasticsearch 8.x: Master-eligible nodes use node.roles: [master, ...]. The one-time cluster.initial_master_nodes setting names the initial voting nodes.

  • OpenSearch 3.x: The equivalent role is cluster_manager, and the current initial-bootstrap setting is cluster.initial_cluster_manager_nodes.

  • Both products use discovery.seed_hosts to locate initial peers.

  • Treat initial-bootstrap settings as temporary. Remove them after stable cluster formation so an unrelated cluster is never bootstrapped from incomplete state.

Failure Modes

  • Master process fails: Other master-eligible nodes elect a replacement, usually causing a brief interruption to cluster-state operations.

  • Quorum is lost: No node can become master. Index creation and deletion, mapping changes, shard reallocation, and other metadata operations stop. Searches and writes against existing shards may continue temporarily.

  • Election timeout: Long garbage-collection pauses, storage latency, or overloaded networking can make a healthy master appear unreachable and trigger unnecessary elections.

  • Split brain: Majority voting, election terms, and the persisted cluster UUID prevent separate masters from accepting conflicting cluster states.

  • Election storms: Repeated master restarts, simultaneous deployments, or unstable networks can repeatedly interrupt operations.

Inspecting Leadership and Planned Master Removal

  • Current master: GET /_cat/master?v shows the elected master in Elasticsearch, and GET /_cat/cluster_manager?v does the same in OpenSearch. GET /_cluster/health?pretty shows overall health but not the master’s identity.

  • Excluding a node before maintenance: POST /_cluster/voting_config_exclusions?node_names=es-2&timeout=30s removes a master-eligible node from the voting configuration so the cluster can still reach quorum while it is stopped.

  • Restoring it: DELETE /_cluster/voting_config_exclusions?node_names=es-2 clears the exclusion after the node has restarted and rejoined, letting it vote again.

  • Important nuance: nodes that leave the cluster gracefully drop out of the voting configuration automatically, but a node that is killed or crashes stays in the voting configuration and can hold quorum hostage. Exclude such nodes explicitly, and only exclude nodes that are genuinely offline or being decommissioned.

Production Best Practices

  • Run three dedicated master-eligible nodes on separate failure domains. Give the master role adequate memory and CPU, but avoid unnecessary data-ingestion workloads on those nodes.

  • Enable TLS for node-to-node communication, restrict transport and HTTP ports to trusted networks, and protect Elasticsearch or OpenSearch APIs with authentication and least-privilege roles.

  • Monitor the elected master, voting configuration, cluster health, unassigned shards, heap pressure, garbage-collection duration, disk watermarks, and election-related logs.

  • Restart or upgrade one master-eligible node at a time. Use voting exclusions for planned removal and verify quorum before proceeding.

  • Keep persistent cluster data on stable storage and ensure sufficient disk headroom. Snapshot repositories protect index data, but snapshots are not a substitute for maintaining master quorum.

9

What is an inverted index in Elasticsearch?

An inverted index is Elasticsearch’s core data structure for fast full-text search. Instead of scanning every complete document for each query, it maps analyzed terms to the documents containing them, enabling Elasticsearch and OpenSearch to retrieve matches efficiently.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How It Works

When a document is indexed, Elasticsearch processes each field according to its mapping:

  1. Normalization and analysis: Text is converted into terms using an analyzer. The standard analyzer lowercases tokens, so Running and runs become running and runs; it does not stem them to run. An analyzer with stemming, such as the English analyzer, can produce the stem run.

  2. Term dictionary: Elasticsearch maintains a dictionary of unique terms, similar to the index at the back of a book.

  3. Postings list: Each term points to the documents containing it. Postings may also contain term frequency, positions, and offsets used for scoring and highlighting.

  4. Lucene storage: The postings, term dictionary, and related structures are stored in immutable Lucene segments within each shard.

Lucene periodically merges smaller segments into larger ones. Merging reduces search overhead and removes data marked for deletion, but it consumes temporary disk space and can temporarily increase I/O. Elasticsearch performs these merges as normal index maintenance rather than rewriting the entire index for every update.

Inverted Index Versus Stored Documents

By default, Elasticsearch stores the original JSON document in _source, but _source is not the primary full-text search structure. The inverted index identifies matching documents, while _source is commonly retrieved afterward to return complete results.

Structure Purpose
Inverted index Finds documents containing analyzed terms
_source Returns the original JSON document by default
Doc values Supports efficient sorting, aggregations, and access to field values
Stored fields Provides selected values directly from the Lucene index

_source can be disabled or configured to exclude individual fields, so it should not be treated as a guaranteed copy of every document.

For exact-value filters, Elasticsearch can use structures such as keyword fields and doc values rather than analyzing the query as free text.

Analysis and Query Behavior

Index-time and search-time analyzers should be compatible, but they do not have to be identical. For example, an index analyzer can produce prefixes or n-grams for a specialized search use case.

  • A match query analyzes its search terms using the field’s search analyzer and therefore supports full-text matching.

  • A term query does not analyze its input and searches for the exact indexed term.

  • A bool query combines conditions such as must, should, filter, and must_not.

  • Filters are commonly placed in filter clauses because matching does not contribute to relevance scoring.

Elasticsearch uses BM25 relevance scoring for typical full-text queries. BM25 considers factors such as term frequency, inverse document frequency, and field length. Filters do not score, while constant_score queries replace variable relevance scores with a constant score.

Mappings determine how fields are indexed. A text field is analyzed for full-text search by default, while a keyword field stores an exact, normally unanalyzed value. Incorrect mappings can cause searches to return unexpected results even when the query itself is correct.

Production and Operational Considerations

  • Sharding: An Elasticsearch index is divided into primary shards, each backed by a Lucene index. A primary shard accepts indexing operations, while replica shards provide redundancy and additional search capacity and can be promoted if a primary fails.

  • Shard sizing: Keep primary shards within a practical size, commonly tens of gigabytes rather than hundreds, and base the count on expected data volume, query concurrency, and recovery time. Oversized shards increase recovery and relocation costs.

  • Cluster health: Missing primary shards make the affected data unavailable and generally produce a red cluster. Unassigned replicas reduce redundancy but do not immediately lose indexed data. Monitor shard allocation, disk watermarks, JVM memory pressure, indexing latency, search latency, and merge activity.

  • Updates and deletes: Deletes are not immediately removed from Lucene segments. Space can be reclaimed through segment merges, but frequent updates can generate considerable temporary I/O.

  • Snapshots: Use repository-backed snapshots for independent backups and disaster recovery. Replicas provide redundancy and availability, but because they contain copies of the same data, they are not independent backups.

  • Security: Use TLS for node and client communication, enforce authentication and role-based access, and avoid exposing Elasticsearch or OpenSearch directly to untrusted networks.

  • Upgrades: Follow the supported upgrade path and check compatibility requirements for plugins and client versions. Major-version upgrades may require migration steps rather than a simple in-place restart.

Elasticsearch and OpenSearch

Elasticsearch and OpenSearch both use Apache Lucene, so their inverted-index design, analyzers, postings, segment merging, BM25 scoring, and shard architecture are closely aligned. The main practical differences lie in distribution packaging, supported versions, plugins, administrative APIs, security tooling, and cluster-management features rather than the fundamental inverted-index concept.

10

What are Lucene segments and how does segment merging work?

Lucene segments are immutable collections of indexed data that form the physical storage beneath an Elasticsearch or OpenSearch shard. Segment merging combines selected segments into larger, more compact segments, removes eligible deleted documents, and reduces the overhead of searching many small segments.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

Segment Anatomy

Each shard is backed by a Lucene index containing one or more immutable segments. Elasticsearch and OpenSearch search the shard’s live segments together through a Lucene reader; individual segments are not exposed as separately queryable indexes.

A segment can contain:

  • Term dictionaries and postings: Maps analyzed terms to the documents containing them.

  • Stored fields: Values retained in the Lucene index for retrieval.

  • Doc values: Column-oriented values used for sorting, aggregations, and access in scripts.

  • Norms: Document-length and relevance information used by scoring.

  • Point values: Structured numeric, date, and other values used for range-related queries.

  • Field metadata: Information describing indexed fields and their encodings.

Deletion information records which documents have been logically deleted without immediately rewriting their original segment. Elasticsearch and OpenSearch also normally retain the original JSON document in _source, which is separate from Lucene stored fields.

Refreshes, Updates, and Deletions

New documents are first buffered in memory. A refresh publishes those changes to a new near-real-time search view, normally by writing one or more segments. It does not necessarily establish a durable Lucene commit point; flushing performs the commit operation.

An update is implemented as a deletion of the previous document version plus insertion of the new version. Consequently, deleted documents can continue occupying disk space until a later merge removes them. Merge policies can retain certain soft-deleted documents when required by a retention or history feature.

How Segment Merging Works

Lucene’s merge policy periodically selects segments that satisfy its size, segment-count, and merge-work limits. The merge process:

  1. Reads the selected segments.

  2. Combines their indexed data and field metadata.

  3. Omits eligible deleted documents.

  4. Writes one or more replacement segments, subject to segment-size limits.

  5. Publishes the new index view and commit state.

  6. Reclaims obsolete segment files after they are no longer needed by an open reader.

Merging does not change the shard count or the logical document count. It can reduce segment count, file-handle pressure, deleted-document overhead, and per-segment search costs, but it causes write amplification and temporarily requires additional disk space.

Automatic merging does not combine every segment immediately. Current Elasticsearch and OpenSearch merge policies balance segment-count tiers, minimum and maximum segment sizes, and limits on how many segments can be merged at once. This avoids rewriting the entire index for every small change.

Elasticsearch 8.x and OpenSearch 2.x

Area Elasticsearch 8.x OpenSearch 2.x
Storage engine Apache Lucene Apache Lucene
Segment behavior Immutable segments with automatic merging Immutable segments with automatic merging
Inspection APIs _segments, _cat/segments, and node statistics _segments, _cat/segments, and node statistics
Forced merge API _forcemerge _forcemerge
Lifecycle automation Index Lifecycle Management (ILM) Index State Management (ISM)
Main administrative difference ILM and Elasticsearch-specific operations ISM and OpenSearch-specific operations

The underlying Lucene behavior is closely related because both systems use Lucene. Their principal differences concern administration, lifecycle automation, security, and observability rather than segment construction or merging.

Operational Guidance

  • Prefer automatic merging: Force merging a busy write index can monopolize disk bandwidth and increase indexing and search latency.

  • Use _forcemerge exceptionally: It can be justified for a largely read-only index with substantial delete overhead, but it should not be a routine disk-space or performance remedy.

  • Monitor merge pressure: Track segment counts, deleted-document ratios, merge activity, disk latency, and shard size. Replicas perform merges independently, so ingestion can create background load on multiple nodes.

  • Account for temporary growth: Merged segments must be written before obsolete files can be reclaimed, so available disk must exceed the shard’s steady-state requirement.

  • Size shards deliberately: A common starting range is approximately 10–50 GB per shard, but recovery-time objectives, retention, query load, and relocation cost determine the appropriate size.

  • Balance allocation across nodes and fault domains: A replica should not be concentrated with other copies on the same failure-prone nodes.

  • Investigate delete-heavy indices: A high deleted-document ratio warrants analysis, but it does not automatically justify a forced merge.

11

What is Elasticsearch's refresh interval and why is it near real-time?

The refresh interval determines how long Elasticsearch may defer exposing newly indexed documents to search. This is why Elasticsearch provides near real-time (NRT) search: successful writes can normally become searchable within seconds instead of requiring a separate index rebuild or long commit cycle.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Near Real-Time Search Works

When a document is indexed, Elasticsearch coordinates the write with the document’s primary shard and its replica copies. The operation is recorded in each shard’s transaction log, or translog, providing durability independently of the search refresh cycle.

During a periodic refresh, the shard reopens its searcher over the Lucene segments containing its latest operations. An indexing buffer may be flushed into a new Lucene segment, although a refresh does not necessarily create a segment every time. Search requests can then see the newly indexed operations. Older immutable segments remain searchable and may later be merged into larger segments.

Refresh therefore marks the boundary between a successful index operation and visibility through the normal search API. A document can already be durable and retrievable through the real-time GET API before it appears in search results. The real-time GET path can inspect the translog and does not depend on the normal refresh cycle.

Refresh Interval and Defaults

  • Default interval: 1s for an index unless its settings or an index template specify another value.

  • Per-index setting: index.refresh_interval controls the interval for each index and can be changed dynamically.

  • Automatic refresh: Elasticsearch refreshes an index approximately when the configured interval elapses and changes require visibility. The interval is not a strict upper bound because scheduling and cluster activity can delay a refresh.

  • Disabled automatic refresh: Setting index.refresh_interval to -1 disables automatic refreshes. New documents remain invisible to normal searches until an explicit refresh occurs.

  • Explicit refresh: POST /index/_refresh refreshes the index and its shards immediately.

The refresh interval controls search visibility, not durability. Durability is determined primarily by translog synchronization, acknowledged writes, and replica durability. Increasing the refresh interval therefore does not make successfully acknowledged writes less durable.

Search and Write Parameters

The index API supports a refresh query parameter for write operations:

  • refresh=false: Do not wait for or force a refresh. This is the default and avoids refresh overhead in the write request.

  • refresh=wait_for: Wait for the next scheduled refresh to make the write visible to search.

  • refresh=true: Force an immediate refresh and wait for it to complete before returning.

wait_for is usually the best compromise for applications requiring read-after-write visibility. refresh=true is convenient for low-volume workflows but is unsuitable for a high-throughput request path because each write can force additional segment work.

Operational Trade-Offs

Configuration Search visibility Operational impact Typical use
1s Approximately one second Moderate refresh and indexing overhead General-purpose applications
5s or 10s Several seconds Fewer refreshes and smaller segments High-volume ingestion with relaxed consistency
-1 plus explicit refresh Operator-controlled Lowest refresh overhead Bulk restore or controlled loading
refresh=true After the forced refresh completes Highest per-request indexing cost Tests and low-volume workflows

Frequent refreshes can produce more small segments, filesystem activity, and subsequent merge work. An interval that is too long can cause stale search results, delayed aggregations and dashboards, and application-level read-after-write consistency failures.

Elasticsearch and OpenSearch

Elasticsearch 8.x and OpenSearch 2.x/3.x use the same core NRT model based on Lucene segments and periodic shard refreshes. The index.refresh_interval setting, _refresh endpoint, and write-level refresh parameter behave similarly.

There is no material difference between the products for this particular mechanism. Their newer distributed search, vector, and lifecycle features differ, but they do not change the basic refresh semantics described here.

Best Practices and Common Pitfalls

  • Keep the default 1s interval unless ingestion volume or consistency requirements justify changing it.

  • Prefer refresh=wait_for over refresh=true when a request must confirm search visibility.

  • Avoid refresh=true in high-throughput request paths.

  • Remember that normal search queries and aggregations only include documents visible after a refresh.

  • Do not confuse refresh with translog durability or replica synchronization.

  • During bulk restores, temporarily setting index.refresh_interval to -1 can improve throughput, but refresh the index before exposing it to search traffic.

  • Monitor indexing latency and failures, translog growth, rejected writes, shard allocation, segment counts, and merge pressure.

12

What is the Elasticsearch translog?

The translog is Elasticsearch’s per-shard write-ahead log. It records index and delete operations so acknowledged changes can be recovered after a node crash, even if Lucene has not yet committed them to a durable segment.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How the Translog Works

Each shard has its own translog, including primary and replica shards. It is separate from the replication protocol: replication transports operations between shard copies, while each shard’s translog provides local durability.

A typical indexing sequence is:

  1. Elasticsearch appends the operation to the shard’s translog.

  2. The operation is applied to Lucene’s in-memory index.

  3. Under the default request durability mode, the translog is synced before the operation is acknowledged.

  4. A Lucene refresh makes the operation searchable, but does not necessarily commit it durably.

  5. A Lucene flush commits segments and records a translog checkpoint.

  6. Superseded translog operations are cleaned up later, subject to retention requirements.

The checkpoint identifies the first operation not represented by the committed Lucene state. It allows Elasticsearch to determine which translog entries must be replayed during recovery.

Durability and Recovery

After a node restart or crash, Elasticsearch restores the shard from its last Lucene commit and replays applicable translog operations. This can temporarily increase recovery time and disk activity, but it minimizes acknowledged-write loss.

The important durability settings distinguish when the translog is synced:

  • Request: Sync before acknowledging the operation; this is the safest normal setting.

  • Index: Sync when Lucene flushes, reducing some I/O but increasing the potential loss window.

  • Async: Avoids explicit syncing and provides weaker durability guarantees.

Changing durability settings is appropriate only when the application has explicitly accepted a possible loss of recently acknowledged writes. Hardware and filesystem behavior must also honor flush requests correctly for the durability guarantee to hold.

Refresh, Flush, and Checkpoint

These mechanisms solve different problems:

Mechanism Purpose
Refresh Makes recent operations visible to search
Flush Commits Lucene segments and advances translog durability state
Checkpoint Records the translog position associated with the committed Lucene state
Snapshot Creates a recoverable backup of committed index data

A flush is not required for every write, and manually invoking it is usually unnecessary. For example, POST /my-index/_flush is valid, but routine flushing can create expensive segment commits and should not be used as a substitute for normal recovery mechanisms.

Production Failure Modes

  • Disk pressure: A full or nearly full data disk can stop indexing, trigger watermarks, and prevent shard recovery.

  • Slow storage: High fsync latency directly increases indexing latency, especially with small requests.

  • Translog growth: Long-running flushes or recovery problems can leave large translog files that consume disk space.

  • Filesystem or hardware failure: A failed node can lose local translog data unless replicas, snapshots, or remote storage provide another recovery source.

  • Corruption: Damaged translog or Lucene files can prevent a shard from starting, requiring restoration from a replica or snapshot.

  • Data exposure: Translog files may contain document contents and metadata, so data-directory access must be tightly controlled.

Cluster health alone does not fully describe translog pressure. Operators should also monitor disk usage, watermarks, indexing latency, rejected tasks, recovery time, and per-index translog statistics through endpoints such as GET /my-index/_stats/translog.

OpenSearch and Operational Guidance

OpenSearch retains the same core translog, Lucene commit, checkpoint, and recovery model. Newer OpenSearch releases can also use a remote-backed translog configuration, providing an additional remote recovery source rather than relying exclusively on local translog files.

Production guidance:

  • Use snapshots for backups; the translog is not a user-facing backup and raw translog files should never be copied or deleted manually.

  • Use bulk indexing where appropriate to reduce per-operation overhead.

  • Keep replicas for availability, but do not treat replicas as backups.

  • Monitor local and remote-storage capacity before disk watermarks are reached.

  • Protect data directories using filesystem permissions and encryption at rest.

  • Upgrade Elasticsearch and OpenSearch in supported rolling-upgrade sequences and retain the required node data-directory compatibility.

13

What is a mapping in Elasticsearch?

A mapping in Elasticsearch is the schema definition for an index: it declares each field’s data type, how it is indexed and analyzed, and which metadata is stored. It is the equivalent of a table schema in a relational database, but it also controls text analysis, relevance-scoring inputs, and storage trade-offs.

Read full answer

Applies toElasticsearch 8.x (OpenSearch 2.x mappings are largely wire-compatible; differences noted below).

How It Works

  • When you create an index, Elasticsearch needs to know how to interpret each JSON field in incoming documents.

  • With dynamic mapping (the default), Elasticsearch infers types from the first document: a string becomes text plus a keyword multi-field, numbers become long/double, booleans become boolean, and dates matching patterns become date.

  • With explicit mapping, you declare fields upfront via the mappings object in the create-index API.

  • Once a field exists, its type cannot be changed; you must reindex into a new index with the corrected mapping.

  • Mappings are stored in cluster state and replicated to all nodes, so very large mappings slow cluster-state updates.

Key Components

  • Field types: text (full-text, analyzed), keyword (exact value, aggregations/sorting), numeric (long, integer, double, float), date, boolean, object, nested, geo_point, dense_vector (8.x vector search), and sparse_vector in recent 8.x releases.

  • Multi-fields: index the same value multiple ways, e.g. text for search plus a keyword sub-field for aggregations.

  • Analyzers: analyzer, search_analyzer, and normalizer control tokenization at index and query time.

  • Mapping parameters: index (whether searchable), store (stored separately from _source), doc_values (columnar for sorting/aggs), norms (relevance scoring), copy_to (combine fields), ignore_above, null_value.

  • Meta-fields: _source, _id, _index, _routing, _meta.

Dynamic vs Explicit

Aspect Dynamic Explicit
Setup Zero config Upfront declaration
Type safety Risky (wrong inference) Controlled
Production use Avoid for critical fields Recommended
Field explosion Can create hundreds of fields Bounded

OpenSearch Differences

  • OpenSearch 2.x mappings are wire-compatible for most types.

  • OpenSearch vector search uses the knn plugin with knn_vector type and engine settings (nmslib, faiss, lucene); Elasticsearch 8.x uses dense_vector with index: true and HNSW.

  • Some newer Elasticsearch types (e.g. semantic_text in 8.11+) may not exist in OpenSearch.

Best Practices

  • Define explicit mappings in production; disable dynamic mapping or use dynamic: strict to reject unknown fields.

  • Keep field count bounded (default limit 1000 fields per index) to avoid mapping explosion.

  • Use keyword for IDs, statuses, and tags; text only for free-form search.

  • Set ignore_above on keyword fields to avoid indexing huge values.

  • Plan reindexing: changing a type requires a new index plus _reindex plus an alias swap.

  • Monitor mapping size in cluster state and inspect the effective mapping with GET /index/_mapping before changing application code.

14

How do dynamic and explicit mappings differ in Elasticsearch and OpenSearch?

Dynamic mapping lets Elasticsearch or OpenSearch infer a field’s data type when the field first appears in an indexed document. Explicit mapping defines the field’s type and indexing behavior in advance, providing predictable schemas at the cost of additional design and maintenance.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Difference

Aspect Dynamic mapping Explicit mapping
Schema source Inferred from JSON values Defined by the operator
Unknown fields With dynamic: true, added and indexed automatically Determined by the separate dynamic setting, not by explicit properties alone
Type control Depends on observed values and compatibility Deliberately fixed, such as keyword, text, integer, or date
Main benefit Convenience and rapid schema evolution Predictable queries, aggregations, storage, and ingest validation
Main risk Incorrect inference, type conflicts, and mapping explosion More design and maintenance effort

Explicit and dynamic mapping are not mutually exclusive. An index can explicitly define important fields while allowing selected or all remaining fields to be created dynamically.

How It Works

An Elasticsearch or OpenSearch index is divided into primary shards and replica copies. Its mapping is index-level metadata applied to each shard’s Lucene index. The mapping determines:

  • Which fields are searchable, aggregatable, and usable for sorting.

  • Whether strings are analyzed as full text or treated as exact terms.

  • How numbers, dates, IP addresses, and other values are represented.

  • Which values are indexed rather than ignored.

When dynamic mapping encounters a previously unseen field, it inspects the JSON value or values and infers a mapping. Strings are generally mapped to text with a keyword subfield, while JSON numeric and boolean values receive numeric or boolean mappings. Date detection can also recognize date-like strings unless dynamic date detection is disabled.

Once a field has been indexed with a particular type, its type generally cannot be changed in place. For example, a field mapped as long cannot simply be remapped as keyword. A common remediation is to create a correctly mapped index, reindex the data, and perform an alias-based cutover.

Strict, False, and Dynamic Templates

The dynamic mapping parameter controls how fields absent from an explicit mapping are handled. It can be configured at the root or on an object mapping:

  • dynamic: true: Adds unknown fields to the mapping and indexes them. This is the default.

  • dynamic: false: Does not index unknown fields or add them to the mapping, although their values normally remain in _source.

  • dynamic: strict: Rejects documents containing unknown fields, exposing schema drift at ingestion time.

Whether unknown values remain in _source is controlled by the index’s _source setting; it is not determined by the field mapping itself.

dynamic_templates can combine explicit control with dynamic behavior. For example, a template can map matching newly detected strings as keyword rather than the default full-text mapping. This is useful for identifiers, status values, and other fields requiring exact matching. Templates affect dynamically mapped fields, not fields already defined explicitly.

Production Guidance

  • Prefer explicit mappings for fields used in filters, aggregations, sorting, or business reporting.

  • Use index templates to apply mappings before indexes are created, especially for rolled-over time-series indexes.

  • Use dynamic: strict when unexpected fields indicate an upstream schema violation and should cause document rejection.

  • Use dynamic: false when unknown values must be retained but not searched or aggregated.

  • Use dynamic templates carefully so arbitrary user-controlled keys do not create thousands of fields and excessive mapping metadata.

  • Monitor rejected documents, mapping changes, field count, cluster-state pressure, and disk watermarks. Frequent mapping updates can increase cluster overhead.

  • Plan reindexing before ingestion starts where possible; correcting an inferred type later requires a new index, sufficient disk space, and an alias-based cutover.

  • Secure management APIs with TLS and least-privilege identities, such as scoped API keys. Mapping and index-template permissions should be separated from ordinary write access where practical.

Elasticsearch 8.x and OpenSearch 2.x/3.x share the same fundamental mapping model and dynamic behavior. Their security configuration, analyzer plugins, and surrounding platform services differ, but the distinction between inferred and explicitly defined schemas remains the same.

15

How do text and keyword fields differ in Elasticsearch and OpenSearch?

A text field is analyzed into tokens for full-text search, while a keyword field stores each value as one unanalyzed term for exact matching, filtering, sorting, and aggregations. Text fields are typically queried with match; keyword fields are typically queried with term.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Behavior

  • Text field: Runs the configured analyzer during indexing. For example, "Wireless Headphones" might become wireless and headphones, enabling relevance-ranked full-text searches.

  • Keyword field: Stores the entire value as one term without tokenization. "Wireless Headphones" remains one exact term, making it suitable for IDs, status values, product codes, and aggregations.

  • Original value: By default, both field types retain the original value in _source, regardless of how the value was indexed for searching.

  • Normalization: A keyword field can use a normalizer, such as one that lowercases values. This provides case-insensitive exact matching without splitting the value into tokens.

Query and Aggregation Behavior

Use case text field keyword field
Full-text search match analyzes the query text Usually inappropriate for a multi-word search
Exact match term matches an individual indexed token term matches the complete value
Case sensitivity Depends on the analyzer Exact and case-sensitive by default
Filtering Possible, but less suitable Preferred because it avoids analysis
Relevance scoring Usually meaningful with BM25 Usually used in filter or constant_score contexts
Aggregations Often use a .keyword multi-field Can aggregate directly on the keyword field

A term query does not analyze its input. Therefore, searching a text field for "wireless headphones" as one term normally finds nothing because the indexed terms are separate tokens.

Production Considerations

  • Choose by purpose: Use text for human-readable content and keyword for identifiers, enum-like values, exact filters, and aggregations.

  • Use multi-fields: A common pattern is name as text with name.keyword for exact matching. The multi-field adds indexed search data without duplicating the original value in _source.

  • Set deliberate limits: ignore_above prevents exceptionally large values from consuming heap and disk. Values exceeding the limit remain in _source but are not indexed in that field.

  • Control cardinality: Highly unique keyword values increase index and aggregation memory costs. Monitor field cardinality, heap usage, search latency, and disk growth.

  • Treat mapping changes carefully: Changing a field’s type or adding a multi-field to an existing field may require creating a new index and reindexing rather than updating the mapping in place.

  • Plan shard capacity: Count primary shards across the entire cluster, not only one index. Avoid excessive shard counts; large shards increase recovery time, while many small shards consume node heap and cluster-management resources.

  • Monitor cluster health: Track _cluster/health, unassigned shards, pending tasks, search latency, CPU, heap pressure, and disk watermarks. Replicas provide redundancy and can improve read capacity, but they also increase storage and indexing work.

Elasticsearch and OpenSearch

The fundamental text-versus-keyword behavior is the same in Elasticsearch 8.x and OpenSearch 2.x/3.x because both use Lucene-based inverted indexes and similar mapping, analysis, and Query DSL semantics. Available analyzers, plugins, and operational defaults may differ by distribution and version, but the mapping patterns described above work in both.

16

What are multi-fields in Elasticsearch?

Multi-fields are additional mappings beneath a parent field that index the same source value in different ways. They let one logical attribute support full-text relevance search and exact filtering, sorting, or aggregation without duplicating the value in the document source.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How They Work

A multi-field is declared in the parent field’s fields mapping and referenced with dot notation, such as title.keyword. The parent and each sub-field have independent indexed representations, but they are populated from the same source value.

For example, title can be a text field analyzed for full-text search, while title.keyword is a keyword field intended for exact operations. A keyword field does not tokenize its input; it can optionally use a normalizer such as lowercase or asciifolding when case-insensitive matching is required.

Multi-fields are not nested object fields, and a multi-field cannot itself contain further multi-fields. They are also different from field aliases, which point to another field rather than creating another indexed representation.

Common Representations

Representation Typical purpose
text Relevance-oriented full-text search using an analyzer
keyword Exact filters, sorting, and terms aggregations
search_as_you_type Prefix, phrase-prefix, and infix-style search for autocomplete experiences
Normalized keyword Case-insensitive or accent-folded exact matching

Adding a second analyzed text representation is useful when the same value must be searched with different analyzers, such as a standard analyzer and a language-specific analyzer. A search_as_you_type multi-field provides indexed prefix and phrase-prefix structures, but appropriate query types are still required.

Querying

Use an analyzed parent field with queries such as match when token-level relevance is important. Use its keyword sub-field with term when the complete value must match exactly, or reference that sub-field in sorts and aggregations.

Exact matching is case-sensitive by default for keyword fields. A configured normalizer can make indexed terms case-insensitive or normalize accented characters. Aggregations on a keyword multi-field are generally more appropriate than aggregations on analyzed text because analyzed fields produce individual tokens rather than complete values.

Production Trade-offs

  • Indexing cost: Every multi-field creates another indexed representation, increasing indexing work, segment size, heap pressure, and storage use even though _source contains the value only once.

  • Cardinality: A keyword aggregation can consume significant memory when values have high cardinality. Size shards according to the data and aggregation workload, not only document count.

  • ignore_above: Values longer than the configured limit remain in _source but are not indexed or represented in doc values. They can consequently disappear from keyword sorts and aggregation buckets.

  • Mapping changes: A new multi-field can be added to an existing field, after which documents should be reindexed to populate it for older records. A new index may be safer for incompatible or high-risk migrations because reindexing temporarily increases cluster load.

  • Operational rollout: Monitor indexing latency, rejected requests, disk usage, heap pressure, and cluster health during bulk reindexing; use appropriate refresh, slices, and request-throttling controls.

  • OpenSearch differences: Core multi-field syntax is the same in OpenSearch 2.x, but analyzer plugins and supported field types can differ. Confirm availability in the exact OpenSearch distribution before using a type or plugin unavailable in Elasticsearch.

17

What is a mapping explosion in Elasticsearch, and how do you prevent it?

A mapping explosion is uncontrolled growth of an index’s mapping as dynamically discovered field names accumulate. It increases cluster-state and memory pressure and can exhaust the per-index field limit, causing ingestion failures.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.19 and 3.x

How It Happens

With the default dynamic: true, Elasticsearch infers mappings from the first encountered non-null value for each field:

  • Strings normally become a text field with a .keyword multi-field.

  • Numbers, booleans, and recognized date strings become their corresponding types.

  • Arrays use the type inferred from their first non-null element; incompatible later values are rejected.

  • Every distinct property name can become another field, including properties nested inside dynamically mapped objects.

For example, embedding tenant identifiers in names such as tenant_42_status creates a new field for every tenant. Untrusted clients can exploit the same behavior by submitting unique keys.

Production Impact

  • Ingestion failures: The default index.mapping.total_fields.limit is 1000 per index. Further fields can cause documents to be rejected with an HTTP 400 illegal_argument_exception.

  • Cluster-state growth: Mapping changes publish cluster-state updates and increase memory consumed by cluster-manager and data nodes.

  • Search overhead: More Lucene field metadata increases heap usage and can make merges and query execution slower.

  • Difficult recovery: Existing field mappings cannot be removed. Recovery generally requires reindexing into a new index.

  • Misleading health: A cluster may remain green or yellow while an individual index is close to its field limit, so field count requires separate monitoring.

Prevention Choices

Mechanism Correct use and limitation
Explicit properties plus a strict dynamic policy Declares the contract and rejects unknown fields. Explicit properties alone do not disable dynamic mapping.
dynamic: "strict" Rejects documents containing undeclared fields, preventing accidental or malicious expansion. Producers must then conform to the schema.
dynamic: false Retains unknown values in _source without indexing them. It avoids field growth but makes those values unsearchable and unaggregatable.
dynamic: "runtime" Treats undeclared fields as runtime fields. This avoids indexing them but still adds mapping metadata and can consume the total-fields limit.
flattened Maps a known namespace containing arbitrary keys as one field. Leaf values behave like keywords, so numeric range queries and normal text analysis are unavailable.
dynamic_templates Routes dynamically discovered values to appropriate types, commonly avoiding default text plus .keyword duplication. It does not stop unique names from consuming field slots.
Explicit object and nested mappings Bounds repeatable structures when their inner properties are declared. Dynamically mapped object contents can still explode.
Lower total-fields limit Acts as a circuit breaker and fails ingestion early. It is a guardrail, not a substitute for schema design.

Deployment, Monitoring, and Recovery

  • Put mappings and limits in an index or component template, test them with the template-simulation API, and roll indices so they apply before ingestion begins.

  • Validate producer schemas and route arbitrary attributes into designated flattened fields rather than accepting them as top-level properties.

  • Restrict index-template modification and ingestion privileges; mapping limits alone are not an authorization mechanism.

  • Monitor per-index mappings, aggregate field counts and mapping size through cluster-stat APIs, and alert on cluster-manager heap, pending cluster-state tasks, rejected documents, and field-limit errors.

  • Also monitor index.mapping.depth.limit, index.mapping.nested_fields.limit, and index.mapping.nested_objects.limit when using deeply nested documents.

  • Use roughly 10–50 GB per shard as an initial sizing heuristic for many search workloads, then validate it against heap, query load, and recovery time. Shard count does not cure mapping explosion.

  • If a field is missing from an existing index, update that index’s mapping for an additive change or reindex into a correctly mapped index. Template changes affect future indices only.

  • The listed OpenSearch releases support the same core prevention mechanisms and per-index limits. Feature availability differs in older OpenSearch 2.x releases, so upgrade or verify the exact release before relying on flattened or dynamic runtime mappings.

18

Can you change an existing field mapping in Elasticsearch?

Yes, but only within Elasticsearch’s limited update-mapping rules: some parameters of existing fields can be changed, while incompatible changes such as a field type or index-time analyzer require rebuilding the index. The safe production pattern is to create a new index, reindex the documents, validate it, and atomically switch an alias.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 3.x

What Can Change

A mapping controls how fields are indexed and searched. Elasticsearch’s update-mapping API can add new fields and update certain parameters of existing fields, but it cannot replace an existing mapping wholesale or change incompatible field definitions.

Change In-place update? Notes
Add a previously unmapped field Yes Use the update-mapping API.
Change keyword to text No Create a new index and reindex.
Change the index-time analyzer No Existing Lucene postings are not retokenized.
Change index from true to false Yes The reverse, false to true, is not supported.
Remove a field from the mapping No Recreate the index without that field.

The restriction exists because search uses Lucene’s inverted index, whose postings contain the tokenized terms produced when documents were indexed. The JSON in _source may still display the original value, but changing that source does not rebuild the postings. For example, changing a keyword field to text does not automatically turn its existing whole-value tokens into analyzable tokens.

Safe Migration Strategy

  1. Inspect the current mapping and analyzers, including any multi-fields and inherited templates.

  2. Create the destination index with the required mapping and settings. Primary-shard count is fixed at creation, apart from explicitly splitting the index.

  3. Reindex documents from the source into the destination and monitor the reindex task until it completes.

  4. Prevent a consistency gap. For a short migration, pause writes and perform the final reindex. For a longer migration, capture changes with an application-level change stream or CDC mechanism and replay updates and deletes in order; dual writes alone do not propagate deletes reliably.

  5. Refresh the destination, compare document counts, and test representative filters, analyzed searches, sorting, aggregations, and application queries. Equal counts alone do not establish semantic equivalence.

  6. Atomically move the production alias to the destination and retain the old index until rollback and retention requirements are satisfied. If writes can reach the new index, rolling the alias back requires those writes to be synchronized back to the old index as well.

OpenSearch Compatibility

OpenSearch has the same fundamental restriction: existing indexed field mappings cannot be rewritten into incompatible mappings in place. Its mapping, reindex, and alias APIs support the same general migration pattern, and requests can be run through OpenSearch Dashboards Dev Tools.

Use OpenSearch-specific mapping definitions for OpenSearch-specific features such as vector fields and ingest processors. Do not assume Elasticsearch-only plugins, security features, or vector implementations are installed in OpenSearch.

Production Practices

  • Choose shard count from data volume: Primary shards in the approximate 20–50 GB range are a common starting point, not a hard limit. Avoid both tiny shards that create excessive cluster overhead and a few oversized primaries that increase recovery, relocation, and query costs.

  • Check allocation health: A destination should normally be green before traffic moves, meaning its primaries and replicas are allocated. Yellow means replicas are not fully allocated; red means at least one primary is unavailable.

  • Protect capacity and recovery: Monitor disk watermarks, indexing latency, reindex progress, task rejection, and cluster pressure. The destination needs additional disk space for data, replicas, and transient merge overhead.

  • Preserve rollback safely: Take an Elastic or OpenSearch snapshot according to the platform’s backup process. A snapshot provides a recovery point but does not automatically synchronize writes made after the alias switch.

  • Restrict privileges: Give the deployment or migration identity only the index-management, reindex, and alias privileges it requires, and audit those operations because they can alter index structure and write routing.

19

What is zero-downtime reindexing in Elasticsearch and OpenSearch?

Zero-downtime reindexing is the controlled migration of documents into a new index while keeping the search service available, usually by creating the target, copying and validating the data, and then switching readers atomically through an alias. It preserves availability, but without explicit change capture and conflict handling it does not guarantee that concurrent updates and deletes are transferred exactly once.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x–3.x

How Reindexing Works

The _reindex API reads a search snapshot from a source index and uses bulk indexing internally to write documents into a destination index.

  • The source is not modified or deleted automatically.

  • Document IDs are preserved by default unless the request specifies different behavior.

  • The operation is not transactional and does not provide change-data capture.

  • Mappings, analyzers, index settings, lifecycle policies, and aliases are not copied.

  • If the destination does not exist, it can be created with dynamic mappings. Production migrations should instead create it explicitly with reviewed mappings, analyzers, settings, and shard counts.

  • wait_for_completion=false starts a background task and returns a task identifier. Task completion confirms that the copy ended, not that concurrent changes were captured.

Reindexing is commonly used to change incompatible mappings, migrate data to another cluster, split an oversized index, or consolidate indices.

Same-Cluster Migration Pattern

A reliable same-cluster cutover uses this sequence:

  1. Create the destination index with the required mappings, analyzers, settings, and replicas.

  2. Begin dual writing or change capture before the backfill. Make writes version-aware or use equivalent conflict handling so that a stale backfilled document cannot overwrite a newer destination document.

  3. Ensure updates and deletes are propagated; a document-only dual-write path is insufficient.

  4. Refresh the source if newly indexed documents must be visible to the reindex search, then start the backfill and monitor its task.

  5. Replay changes that occurred after the backfill snapshot until the change stream is caught up.

  6. Validate document counts, representative queries, aggregations, mappings, and business invariants. Counts alone can miss stale values and resurrected deleted documents.

  7. Atomically remove the alias from the old index and add it to the new index in one _aliases request.

The alias operation is atomic within a cluster, but that does not make the surrounding dual-write protocol transactional. Resolve or replay in-flight writes before retiring the old path, and make retried operations idempotent.

Cross-Cluster Migration

The same alias procedure cannot atomically switch traffic between clusters because aliases are local to a cluster. Both products support remote reindexing when remote access, allowlisting, TLS, and authentication are configured, but a remote reindex is still a point-in-time copy rather than continuous change synchronization.

For a cross-cluster migration, use database CDC, an event stream, Cross-Cluster Replication (Elasticsearch, or the OpenSearch replication plugin) where applicable, or equivalent change capture to bridge the backfill. Keep the old cluster writable as required, validate replication lag, and then switch clients, routing, or a load balancer. Cross-cluster cutover is not atomic, so retries and a rollback path are essential.

Production Operations

  • Capacity: Reserve disk headroom for the source, destination, snapshots, and temporary merge pressure. Use a practical shard-size heuristic such as roughly 10–40 GB per shard, then adjust it for indexing rate, query load, and recovery-time objectives.

  • Performance: Reindexing consumes CPU, disk I/O, and network bandwidth. Apply appropriate throttling and monitor indexing latency, thread-pool rejection, JVM pressure, disk watermarks, and cluster health.

  • Visibility: Monitor the background task, destination counts, shard distribution, and application error rates. Account for indexing delay and refresh behavior when interpreting counts.

  • Recovery: Configure and test snapshot restoration before the migration. Keep the source available until post-cutover validation and the rollback window complete.

  • Security: Use TLS, authentication, least-privilege roles, encrypted backups, and restricted network access. Never embed credentials in examples.

  • Rollback: Document how to reverse the alias or client route and how to reconcile writes made after the cutover.

Elasticsearch and OpenSearch Differences

Concern Elasticsearch 8.x OpenSearch 2.x–3.x
Reindex and task APIs Native _reindex and task APIs Equivalent _reindex and task APIs
Atomic index aliases Supported Supported
Index lifecycle management Index Lifecycle Management Index State Management
Continuous cross-cluster replication Cross-Cluster Replication Cross-Cluster Replication plugin (leader/follower index replication)

Both products require the same fundamental safeguards: explicit target mappings, ordered change propagation, delete handling, conflict control, validation, capacity planning, and a tested rollback procedure.

20

What are index aliases in Elasticsearch?

An index alias is a secondary name that points to one or more Elasticsearch indices, letting applications query and, when a write index is designated, index through a stable name while the underlying physical indices change. Aliases decouple application code from concrete index names, which is the foundation of zero-downtime reindexing and time-based index rotation.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Aliases Work

  • An alias is cluster-state metadata; it is not a separate index and owns no shards of its own.

  • One alias can point to multiple indices; searches fan out to all backing indices and results are merged.

  • For writes, exactly one backing index must be marked is_write_index: true, or the alias must point to a single index; otherwise indexing through the alias fails.

  • Aliases support per-alias filter and routing values, creating a filtered view or pinning shard routing without duplicating data.

Common Use Cases

  • Zero-downtime reindexing: build products-v2, repoint the products alias, then remove products-v1.

  • Time-based rotation: a logs alias with write index logs-2024.06, rolled over to logs-2024.07 by ILM.

  • Multi-tenant views: a filtered alias exposing only one tenant’s documents.

  • Stable application contract: apps reference orders instead of dated index names.

Aliases vs. Indices

Aspect Index Alias
Stores data Yes No
Owns shards Yes No
Multiple targets No Yes
Write target Itself Requires write index
Filter / routing No Yes

Managing Aliases

Alias changes are made through the _aliases endpoint, typically with an actions array that can add or remove backing indices in one request. For example, during a reindex you can add products-v2 as the write index for products and remove products-v1 in the same atomic update. Searches such as GET /products/_search then fan out to all backing indices, while a filtered alias can be created by adding a filter such as a term query on level: error.

Best Practices and Pitfalls

  • Always designate a single write index when an alias backs multiple indices; otherwise bulk indexing fails.

  • Avoid too many backing indices behind one alias—search fan-out cost grows; use ILM rollover and delete old indices.

  • Alias filters add query overhead and are not a security boundary; use document-level security for tenant isolation.

  • OpenSearch uses the same _aliases API; the security plugin may restrict alias management by role.

  • Monitor cluster state size—thousands of aliases and indices bloat it and slow master operations.

21

What are index templates and component templates in Elasticsearch?

An index template automatically applies settings, mappings, and aliases to newly created indices whose names match its patterns, while a component template is a reusable fragment of that configuration that index templates compose through composed_of.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Definitions and Composition

Template Purpose Key behavior
Index template Selects future indices through index_patterns Automatically applied when a new index matches
Component template Stores reusable settings, mappings, and aliases Does not match indices directly; referenced through composed_of

An index template can contain configuration directly or compose multiple component templates:

  • index_patterns: Index names or patterns such as logs-*.

  • priority: Selects the index template when several templates match. The highest-priority matching template is used.

  • composed_of: Names of component templates to merge into the index template.

  • template: Index settings, mappings, and aliases.

  • version: Optional version used to manage template revisions.

Component templates are useful for standardizing configuration such as:

  • Shard count and replica count

  • Mapping fields and field types

  • Analyzer configuration

  • Refresh intervals

  • Lifecycle or state-management policy references

  • Shared aliases

Matching and Merging Behavior

Only one index template is selected for a new index. If multiple index templates match, Elasticsearch selects the one with the highest priority; templates with equal priority can produce an ambiguous-match error.

When configuration is merged:

  • The index template’s own configuration overrides conflicting component-template configuration.

  • Component templates are evaluated in the order listed in composed_of; a later component template overrides an earlier conflicting value.

  • Mappings are merged, but conflicting field types may prevent index creation.

  • Changing a template affects only newly created indices. Existing indices are not reconfigured retroactively.

A component template cannot independently match indices and has no selection priority. Its purpose is reuse and consistent governance.

Elasticsearch and OpenSearch Differences

Both products support composable index and component templates through _index_template and _component_template APIs.

  • Elasticsearch 8.x: Uses composable templates rather than the removed legacy /_template API. Lifecycle behavior is commonly managed through Index Lifecycle Management and data stream lifecycle policies.

  • OpenSearch 2.x/3.x: Supports composable templates but may also expose legacy template APIs for compatibility. State transitions are commonly managed through Index State Management policies rather than Elasticsearch ILM.

  • Template APIs create or update configuration; they do not create the referenced ILM or ISM policy. Referenced policies and referenced data streams must exist before they are used operationally.

Production Practices and Pitfalls

  • Simulate changes first: Validate the merged settings and mappings with _simulate_index before rolling out a template.

  • Version templates deliberately: Use names such as logs-v1 and logs-v2, then adjust priority or patterns during migration.

  • Avoid overly broad patterns: A pattern such as * can unintentionally affect unrelated indices.

  • Size shards from workload: Do not copy a fixed shard count blindly. Base the count on index size, ingest rate, query concurrency, and recovery-time objectives.

  • Use replicas for availability: The appropriate replica count depends on node count, allocation constraints, and whether the cluster spans failure domains.

  • Review strict mappings: dynamic: "strict" rejects unexpected fields, which protects schema quality but may cause ingestion failures if producers are not controlled.

  • Protect template APIs: Restrict write access to template endpoints and validate changes through CI/CD and approval workflows.

  • Monitor creation failures: Watch cluster health, allocation, rejected writes, and template-related index creation errors after deployment.

22

What is an analyzer in Elasticsearch and OpenSearch?

An analyzer in Elasticsearch and OpenSearch converts text into a stream of normalized tokens used to build and query the Lucene inverted index. It controls how text is tokenized and processed at index time and search time, directly affecting matching behavior, storage size, and query performance.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How It Works

An analyzer consists of three main components:

  • Character filters: Process the raw text before tokenization, such as removing HTML markup or replacing characters.

  • Tokenizer: Splits the text into tokens based on a rule. For example, standard tokenization separates ordinary words, while whitespace uses spaces as boundaries.

  • Token filters: Transform or remove the resulting tokens, commonly lowercasing them, removing stop words, or applying stemming.

The resulting tokens are written into the Lucene inverted index with their postings and, where applicable, position and term-frequency information. During a search, the search analyzer processes the query text so that it is compared using compatible tokens.

Index-Time and Search-Time Analysis

These are separate stages:

  • Index-time analyzer: Normalizes document field values before indexing. The chosen tokens determine what is stored in the inverted index.

  • Search-time analyzer: Normalizes the query text before matching. For a text field, it usually defaults to the field’s index-time analyzer unless search_analyzer is explicitly configured.

The distinction matters for fields such as product names. A document containing "Running Shoes" may be indexed as the tokens running and shoes. A match query analyzes its input similarly, while a term query does not analyze the query value.

Using term for a text field is therefore appropriate only when the query already contains the exact indexed token, such as a keyword-like identifier or a carefully controlled vocabulary value.

Built-in and Custom Analyzers

Common built-in analyzers include:

  • standard: The default for text fields; performs Unicode-aware tokenization and lowercase normalization.

  • simple: Splits on non-letter characters and lowercases tokens.

  • whitespace: Uses whitespace only as the token boundary.

  • keyword: Treats the entire value as one token and performs no normal word tokenization.

Custom analyzers are defined under an index’s analysis settings. A typical custom analyzer specifies a tokenizer and an ordered list of token filters. The order matters because each filter receives the output of the previous component. For example, a custom analyzer might use the standard tokenizer with lowercase and stop filters, reference it through a field’s analyzer and optional search_analyzer settings, and be inspected with the _analyze API before indexing production data.

A keyword field uses a normalizer rather than an analyzer when case-insensitive exact matching is needed. A normalizer produces a single token and supports only a restricted set of filters.

Elasticsearch and OpenSearch Differences

Elasticsearch 8.x and OpenSearch 2.x share the same fundamental analyzer model and provide the common built-in analyzers. Custom analyzer settings and the _analyze API are also broadly equivalent.

The main practical difference is the surrounding platform: OpenSearch may have different plugin availability, security integrations, and version-specific Lucene behavior. Elasticsearch-specific plugins or analyzers should not be assumed to exist in OpenSearch, and compatibility should be verified for the versions being deployed.

Operational Best Practices and Failure Modes

  • Test analysis before indexing: Use _analyze to confirm tokenization, stop-word removal, and normalization behavior.

  • Reindex after changing analysis: Changing an analyzer usually requires a new field, a new index, or a reindex operation because previously indexed tokens are not automatically rewritten.

  • Keep index and search analysis aligned: A mismatch can cause valid-looking queries to miss, especially with stemming, synonyms, or aggressive stop-word filtering.

  • Use keyword for exact values: IDs, codes, and enumerated values generally should not be analyzed as ordinary text.

  • Monitor token volume: Excessive tokenization increases index size, heap pressure, and query cost. Excessive stop-word or synonym expansion can have similar effects.

  • Avoid unnecessary custom processing: Custom analyzers are declarative and do not execute arbitrary application code, but complex token filters can still create significant indexing and search overhead.

  • Plan changes carefully in production: Validate mappings with _mapping, test representative queries, and use aliases or versioned indices to make rollbacks possible.

23

What do Elasticsearch character filters, tokenizers, and token filters do?

Elasticsearch and OpenSearch analyze text through an analysis pipeline containing character filters, a tokenizer, and token filters. They convert human-readable text into normalized tokens used to build and query the inverted index.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Text Analysis Works

Text is processed in a fixed order:

  1. Character filters modify the original text.

  2. A tokenizer splits the modified text into tokens.

  3. Token filters transform, remove, or add tokens.

  4. The resulting tokens are stored in the inverted index.

For example, analyzing Running with the standard tokenizer and a lowercase filter produces the token running.

String fields mapped as text use the standard analyzer by default unless the mapping specifies another analyzer. Index-time and search-time analysis can differ through analyzer and search_analyzer.

Character Filters

Character filters operate on the entire input string before tokenization. They are useful for cleaning text or mapping inconsistent representations to a common form.

Common filters include:

  • html_strip: Removes HTML tags from the source text.

  • mapping: Replaces characters according to configured character mappings.

  • pattern_replace: Replaces text matching a regular expression.

A practical use is normalizing a product code such as ABC-123 into ABC 123 before tokenization.

Character filters affect the indexed tokens, but they do not modify the document’s original _source. Also, html_strip is not a complete HTML sanitizer or security boundary.

Tokenizers

A tokenizer determines where token boundaries occur. Its output determines the basic token shape that later filters can modify.

Common choices include:

  • standard: Splits text using Unicode word-boundary rules and is suitable for most general-purpose fields.

  • whitespace: Splits only on whitespace, making it useful when punctuation must be preserved.

  • keyword: Treats the entire input as one token, appropriate for identifiers, SKU codes, and exact values.

  • simple_pattern: Splits on a fixed delimiter pattern.

  • pattern: Splits using a more complex regular expression.

The tokenizer choice is significant. A keyword tokenizer treats example.com as one token, while standard normally produces example and com.

Token Filters

Token filters process the tokenizer’s output. They can remove, rewrite, or generate tokens.

Frequently used filters include:

  • lowercase: Normalizes tokens to lowercase.

  • asciifolding: Converts accented characters to ASCII equivalents.

  • stop: Removes common stop words.

  • stemmer: Reduces words to a stem, such as running to run.

  • ngram: Creates overlapping n-grams for substring matching.

  • edge_ngram: Creates prefixes for autocomplete.

  • synonym: Maps terms to equivalent or related terms.

  • word_delimiter_graph: Splits compound words while preserving graph relationships for phrase matching.

Filter order matters. For example, a lowercase filter placed after a stemmer produces different results from one placed before it.

Choosing an Analyzer

Requirement Recommended approach
General full-text search standard analyzer
Exact codes or identifiers keyword analyzer or keyword tokenizer
Case-insensitive matching standard analyzer, which already includes lowercase normalization
Autocomplete edge_ngram at index time and a suitable search analyzer
Language-specific stemming Language-specific stemmer or plugin analyzer
Synonym matching synonym or synonym_graph with careful test cases

Production Trade-offs

  • Index/search mismatch: If indexing uses one analyzer and searching uses another, valid documents may fail to match. Use search_analyzer deliberately rather than relying on defaults.

  • Query behavior: match queries analyze their search text, while term queries generally do not. A term query must target the exact token stored in the index.

  • Storage cost: ngram, edge_ngram, and aggressive stemming increase index size and CPU usage.

  • Synonyms: Synonym expansion can increase query cost and create ranking differences. Version synonym sets carefully and test phrase behavior.

  • Plugin differences: Core analysis is similar in both products, but language and third-party analyzer plugins differ. Elasticsearch and OpenSearch also have different plugin compatibility and release lifecycles.

  • Operational validation: Test analyzer output with the _analyze API and monitor indexing latency, query latency, and index size after changing mappings. Existing fields may require reindexing because mapping analyzer changes are not applied retroactively.

24

How do you create a custom analyzer in Elasticsearch?

A custom analyzer in Elasticsearch defines how text is converted into tokens before indexing or searching. You compose it from optional character filters, one tokenizer, and zero or more token filters, then reference it from an index mapping or use it through the _analyze API.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x

How It Works

The analysis pipeline runs in a fixed order:

  1. Character filters modify the original text, such as removing HTML or replacing patterns.

  2. The tokenizer splits the text into tokens, for example standard, whitespace, or keyword.

  3. Token filters transform or remove tokens, such as lowercasing, removing stop words, applying stemming, or folding accents.

The analyzer used at index time creates the terms stored in the Lucene inverted index. A search-time analyzer tokenizes the query similarly so that Elasticsearch can look up matching terms.

A custom analyzer is defined in an index’s settings.analysis section. It is not normally a cluster-wide setting. A normalizer is different: it supports only a restricted set of filters, such as lowercase and asciifolding, and is useful for exact-match fields such as product codes or usernames.

Configuration Example

The following creates an index with:

  • A product_search custom analyzer that removes a leading SKU- prefix, lowercases terms, folds accents, and removes English stop words.

  • A product_normalizer for an exact-match keyword field.

PUT /products-v1
{
  "settings": {
    "analysis": {
      "char_filter": {
        "strip_product_code": {
          "type": "pattern_replace",
          "pattern": "^\\s*(?:SKU[-_])?",
          "replacement": ""
        }
      },
      "filter": {
        "product_stop": {
          "type": "stop",
          "stopwords": "_english_"
        }
      },
      "analyzer": {
        "product_search": {
          "type": "custom",
          "char_filter": ["strip_product_code"],
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "product_stop"]
        }
      },
      "normalizer": {
        "product_normalizer": {
          "type": "custom",
          "char_filter": [],
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "product_search",
        "search_analyzer": "product_search"
      },
      "sku": {
        "type": "keyword",
        "normalizer": "product_normalizer"
      }
    }
  }
}

The analyzer can be tested without indexing data:

POST /products-v1/_analyze
{
  "analyzer": "product_search",
  "text": ["Café Running Shoes"]
}

The response contains the token stream produced by the configured analyzer. Test representative production values, including punctuation, Unicode characters, whitespace, and malformed input, before rolling the mapping out broadly.

Production and Operational Guidance

  • Keep index-time and search-time analysis aligned: If name is indexed with product_search, use a compatible analyzer for searches. A match query uses the field’s search_analyzer, while a term query is not analyzed.

  • Plan changes with reindexing: Changing analysis settings does not rewrite tokens already stored in existing documents. In production, create a new index with the desired settings and reindex into it, usually through an index template and a controlled migration.

  • Avoid accidental token explosion: Filters such as ngram, edge_ngram, and synonym filters can create many terms and increase index size, heap usage, and query cost. Apply them only to fields that need them.

  • Use a normalizer for exact identifiers: A keyword field with a normalizer provides consistent matching for case-insensitive or accent-insensitive identifiers without the complexity of a full text analyzer.

  • Use templates and versioned indices: Define analyzers in index templates so that new indices receive identical settings. Version the index name, for example products-v1, and use aliases to switch application traffic during migrations.

  • Protect the cluster: Require authenticated, authorized users to create or modify indices. Limit write access to ingestion pipelines and use TLS for traffic between Elasticsearch nodes, clients, and Kibana.

  • Monitor the impact: Track indexing latency, search latency, JVM heap pressure, rejected requests, shard count, and cluster health when introducing analysis changes. Snapshot the data before a large reindex and verify restore procedures regularly.

Elasticsearch and OpenSearch Differences

The core analyzer syntax shown above is compatible with Elasticsearch 8.x and OpenSearch 2.x/3.x because both use Lucene-based analysis. Plugin-provided analyzers and filters can differ between distributions, so verify that any custom component is available in the specific OpenSearch version and deployment.

In managed Elasticsearch offerings, use the provider’s supported reindex, composable template, and monitoring features rather than relying on ad hoc index creation. OpenSearch deployments should similarly account for the distribution’s plugin model, IAM configuration, and index lifecycle tooling.

Common Pitfalls

  • Defining an analyzer but not referencing it: The analyzer remains unused unless a field mapping, a query, or another analyzer references it.

  • Confusing match and term: match analyzes the query text; term searches for an exact indexed term.

  • Changing settings without reindexing: Existing inverted-index terms are not retroactively transformed.

  • Using aggressive filters by default: Stemming, synonyms, and n-grams can improve recall but may reduce precision or increase operational cost.

  • Testing only simple ASCII text: Real requests often contain Unicode, mixed case, punctuation, and inconsistent whitespace, which can expose unexpected tokenization behavior.

25

How do you use Elasticsearch's _analyze API to test analyzers?

The _analyze API runs text through an analyzer and returns the resulting tokens, offsets, types, and positions. It lets you validate tokenization and filters—such as lowercasing, stemming, stop-word removal, and character folding—without indexing the test text.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x

How It Works

The analyzer can be selected in several ways:

  • analyzer: Uses a built-in analyzer such as standard, whitespace, or keyword.

  • tokenizer and filter: Tests a tokenizer and filter chain directly without creating an index.

  • field: Uses the analyzer configured for a field in an existing index.

  • Named custom analyzer: Tests an analyzer defined in an index’s settings.analysis configuration.

Set "explain": true to inspect token metadata such as offsets, positions, token types, position lengths, and the tokenizer or filters that processed each token.

Code Example

Test the built-in standard analyzer:

POST /_analyze
{
  "analyzer": "standard",
  "text": "The Quick Brown Fox",
  "explain": true
}

Test a tokenizer and filter chain directly:

POST /_analyze
{
  "tokenizer": "standard",
  "filter": ["lowercase", "asciifolding"],
  "text": "Café, CRÈME & Espresso",
  "explain": true
}

The standard tokenizer does not emit punctuation as tokens, lowercase normalizes case, and asciifolding converts accented characters to ASCII equivalents where possible. The token café therefore normally becomes cafe.

Testing a Custom Analyzer and Mapped Field

A named custom analyzer must first be defined in the target index’s analysis settings. This index defines article_text and attaches it to the body field:

PUT /articles
{
  "settings": {
    "analysis": {
      "analyzer": {
        "article_text": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "body": {
        "type": "text",
        "analyzer": "article_text"
      }
    }
  }
}

Test the named analyzer:

POST /articles/_analyze
{
  "analyzer": "article_text",
  "text": "Café & Crème",
  "explain": true
}

Test the analyzer configured on the mapped field:

POST /articles/_analyze
{
  "field": "body",
  "text": "Café & Crème"
}

Named-analyzer and field-based tests require the index to exist. The _analyze request does not index the test text, although creating the test index modifies cluster state.

Interpreting Results

  • Token: The normalized output that can be stored in the inverted index.

  • start_offset and end_offset: Character positions in the original input; these remain useful when a character filter changes token length.

  • position: The token’s position in the token stream, which matters for phrase matching.

  • type: The token category, such as <ALPHANUM> or <NUM>.

  • positionLength: The number of positions occupied by a token. It is normally 1 and matters mainly for filters that deliberately emit multi-position tokens. Stemming normally preserves one position, while synonym filters can emit several terms at the same position.

Test representative production content rather than only a simple sentence: Unicode text, mixed case, punctuation, numbers, HTML-like input, multilingual terms, and domain-specific abbreviations.

OpenSearch Differences

OpenSearch 2.x and 3.x support the same core _analyze workflow, including built-in analyzers, direct tokenizer/filter chains, named custom analyzers, mapped fields, and explain.

  • Analysis plugins must be installed consistently across the cluster so shards produce the same tokens.

  • Bundled analysis components and plugin names may differ from Elasticsearch 8.x.

  • Validate plugin analyzer names against the exact OpenSearch release rather than assuming Elasticsearch plugin compatibility.

Operational Best Practices

  • Use a dedicated development or staging index for analyzer definitions; do not create disposable test indices on a busy production cluster.

  • Treat analyzer input as potentially sensitive. Avoid sending credentials, customer records, or regulated data to shared or heavily logged environments.

  • Confirm that the tested analyzer is attached to the intended mapping field and distinguish the index-time analyzer from any search_analyzer.

  • Changing the analyzer for an existing field does not rewrite existing inverted-index terms. Create a new index and reindex when the indexed token stream must change.

  • Monitor _analyze latency and errors, and restrict access according to the environment’s security policy.

26

How do you configure synonym expansion for Elasticsearch 8.x?

Synonym expansion in Elasticsearch is implemented through analyzer token filters, not as a separate search service. Use synonym_graph for search-time, multi-word rules, and use updateable synonym sets when product or business terminology changes without requiring a full reindex.

Read full answer

Applies toElasticsearch 8.10+ for updateable synonym sets; Elasticsearch 8.x and OpenSearch 2.x/3.x for static synonym filters

How Synonyms Work

Elasticsearch analyzes text both when indexing documents and when processing many queries. A synonym filter changes the token stream before Lucene creates or searches the inverted index.

  • Equivalent rules: phone, smartphone normally expand in both directions.

  • Directional rules: phone => mobile phone expands only from the left side.

  • Multi-word rules: Use synonym_graph instead of the basic synonym filter. It preserves alternative token graphs and works better with phrase queries.

  • Search-time expansion: Put the filter in search_analyzer. Query variants such as smartphone can match documents indexed with mobile phone without changing stored documents.

  • Index-time expansion: Put the filter in the field’s indexing analyzer. Synonyms become part of the indexed tokens, so changing the rules normally requires a reindex.

Lucene segments are immutable. Therefore, changing a search-time analyzer does not rewrite existing segments, while changing an index-time analyzer does.

Query and Relevance Behavior

The query type determines whether synonym expansion occurs:

  • match, match_phrase, and other analyzed queries use the field’s search_analyzer and can expand synonyms.

  • term queries are not analyzed and do not expand synonyms.

  • A match query may search both smartphone and mobile phone, depending on the configured operator.

  • BM25 scores can change because synonym expansion affects query terms, term frequencies, and document statistics. Broad equivalence classes can reduce precision.

  • Use bool with a filter clause when synonyms are needed for matching but should not influence the score.

Synonyms affect lexical matching. They do not modify dense vector embeddings or automatically provide semantic-vector equivalence.

Code Example

Create a searchable synonym analyzer using synonym_graph:

PUT products
{
  "settings": {
    "index": {
      "analysis": {
        "filter": {
          "product_synonyms": {
            "type": "synonym_graph",
            "synonyms": [
              "mobile phone, smartphone",
              "laptop, notebook computer"
            ]
          }
        },
        "analyzer": {
          "product_search_analyzer": {
            "tokenizer": "standard",
            "filter": [
              "lowercase",
              "product_synonyms"
            ]
          }
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard",
        "search_analyzer": "product_search_analyzer"
      }
    }
  }
}

A query using match will use the configured search analyzer:

GET products/_search
{
  "query": {
    "match": {
      "name": "smartphone"
    }
  }
}

For frequently changing rules, create an updateable synonym set:

PUT _synonyms/product-terms
{
  "synonyms_set": [
    {
      "id": "1",
      "synonyms": "mobile phone, smartphone"
    },
    {
      "id": "2",
      "synonyms": "laptop, notebook computer"
    }
  ]
}

Then define the analyzer filter as:

{
  "product_synonyms": {
    "type": "synonym_graph",
    "synonyms_set": "product-terms",
    "updateable": true
  }
}

Updateable synonym sets are intended primarily for search analyzers, so terminology changes do not require rewriting indexed documents.

Production Practices and OpenSearch Differences

  • Prefer search-time expansion: It allows operational changes without reindexing. Use index-time expansion only when synonym tokens must be stored for performance or specific scoring behavior.

  • Use synonyms_path carefully: The file must be UTF-8 and available on every node that may execute the analyzer. After changing it, reload search analyzers, for example with POST products/_reload_search_analyzers.

  • Manage versions deliberately: Use aliases and versioned indices with reindexing through index lifecycle management rather than modifying mappings in place.

  • Test the analyzer: Use POST products/_analyze with the custom analyzer to verify how a term expands before relying on it in production.

  • Monitor production health: Track query latency, indexing latency, rejected requests, shard allocation, disk watermarks, and merge pressure. Synonym changes can suddenly increase the number of query terms and affect BM25 scores and latency.

  • Protect the synonym source: Restrict synonym-set administration with least-privilege roles. Do not allow arbitrary user input to create synonym rules, because malformed or overly broad rules can affect relevance and consume resources.

  • OpenSearch: The basic synonym filter and synonyms_path are the most portable options. Recent OpenSearch releases also provide synonym-set APIs, but privileges and supported filter options should be verified against the installed release. OpenSearch’s synonym rules generally have the same analyzer and query-time behavior as Elasticsearch.

27

How should Elasticsearch serve autocomplete using n-grams or completion fields?

Implement autocomplete either by indexing prefix tokens with an edge n-gram analyzer or by storing suggestions in a dedicated completion suggester field. Edge n-grams support flexible partial-token matching, while completion suggesters provide prefix lookup optimized for ordered suggestion lists.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Edge n-Grams Work

An edge n-gram analyzer generates prefixes for each indexed token. For example, Elasticsearch with min_gram: 2 and max_gram: 20 produces tokens such as el, ela, elas, and longer prefixes.

  • These tokens are stored in Lucene’s inverted index, allowing ordinary match queries or appropriately cased prefix queries to find partial terms.

  • Use the edge n-gram analyzer only at index time and a normal analyzer such as standard at search time.

  • Keep min_gram reasonably high, commonly 2 or 3, because a minimum of 1 creates excessive index entries.

  • The difference between max_gram and min_gram must not exceed the index’s index.max_ngram_diff setting, whose default is 1. Configure it explicitly when using a larger range.

  • Multi-word queries can use "operator": "and" to require every analyzed query token to match. This does not enforce phrase adjacency.

  • BM25 scores the matching documents, but it does not provide explicit suggestion ordering.

This approach is useful when autocomplete must match document fields, support multiple analyzed fields, or combine partial matching with filters.

How Completion Suggesters Work

A completion field stores suggestion inputs in a finite-state transducer, or FST. The suggester traverses that structure by prefix and returns matching suggestion values rather than returning the indexed documents themselves.

  • Use a separate completion field when the result should be a controlled title, product name, or query suggestion.

  • weight controls ordering; larger values rank first.

  • Use contexts to restrict suggestions by tenant, category, language, or another request attribute. Contexts must be configured in the completion mapping and attached to suggestion inputs.

  • Completion data is distributed across shards. Elasticsearch and OpenSearch query the relevant shards and merge their suggestions.

  • Newly indexed suggestions become searchable after the index refresh, so results are near-real-time rather than immediately consistent.

Choosing an Approach

Consideration Edge n-grams Completion suggester
Stored representation Prefix tokens in an inverted index Suggestion inputs in an FST
Result Matching documents Ranked suggestion values
Ordering BM25 relevance Explicit weight ordering
Filtering Normal query filters and aggregations Contexts; ordinary query filters do not restrict suggestions
Index cost Can expand the index significantly FST and metadata add write and storage cost
Best fit Search-as-you-type over document fields Controlled suggestion catalogs

Both techniques use compatible mapping and request bodies in Elasticsearch 8.x and OpenSearch 2.x/3.x. Run them in Kibana Dev Tools for Elasticsearch and OpenSearch Dashboards Dev Tools for OpenSearch; cluster-management APIs differ between the products.

Code Example

Create an edge n-gram index and search it using Dev Tools syntax:

PUT autocomplete
{
  "settings": {
    "index": {
      "max_ngram_diff": 18
    },
    "analysis": {
      "filter": {
        "autocomplete_filter": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20
        }
      },
      "analyzer": {
        "autocomplete_index": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "autocomplete_filter"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "autocomplete_index",
        "search_analyzer": "standard"
      }
    }
  }
}

PUT autocomplete/_doc/1?refresh=wait_for
{
  "title": "Elasticsearch cloud"
}

GET autocomplete/_search
{
  "query": {
    "match": {
      "title": {
        "query": "ela",
        "operator": "and"
      }
    }
  },
  "_source": ["title"]
}

For a completion suggester:

PUT suggestions
{
  "mappings": {
    "properties": {
      "suggest": {
        "type": "completion"
      },
      "title": {
        "type": "text"
      }
    }
  }
}

PUT suggestions/_doc/1?refresh=wait_for
{
  "title": "Elasticsearch Cloud",
  "suggest": {
    "input": ["elasticsearch cloud", "elasticsearch consulting"],
    "weight": 10
  }
}

GET suggestions/_search
{
  "suggest": {
    "title-suggest": {
      "prefix": "ela",
      "completion": {
        "field": "suggest",
        "size": 5,
        "skip_duplicates": true
      }
    }
  }
}

Production Practices

  • Shard sizing: Begin with primary shards around 10–30 GB for large searchable indices, then benchmark. Avoid both many tiny shards and excessive shard counts on each node.

  • Capacity: Measure index growth and suggestion-document size before increasing replicas. Replicas improve availability but increase storage and indexing work.

  • Monitoring: Track suggest-request latency and errors, indexing latency, rejected requests, JVM heap and GC pressure, disk watermarks, shard count, and cluster health.

  • Failure modes: Edge n-grams can cause index bloat and overly broad matches. Indexing every distinct query log as a separate completion input can also grow the catalog rapidly; normalize inputs and enforce cardinality limits.

  • Security: Require authentication and TLS, rate-limit autocomplete requests, and avoid exposing unrestricted completion endpoints to untrusted clients.

  • Changes and upgrades: Analyzer or mapping changes do not rewrite existing Lucene segments automatically. Reindex into a versioned index, verify results, then switch traffic through an alias. Use snapshots for important data, although derived autocomplete indexes can often be rebuilt from the source of truth.

28

How does the Elasticsearch Query DSL work?

The Query DSL is Elasticsearch’s JSON-based request language for searching, filtering, scoring, sorting, and aggregating documents. OpenSearch supports a largely compatible DSL, although plugins, administrative APIs, and newer capabilities such as vector search can differ.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x; Elasticsearch k-NN from 8.12 and OpenSearch with the k-NN plugin

Request Structure and Query Model

A search request can contain several top-level fields:

  • query: Selects matching documents and, depending on the query type, calculates their relevance scores.

  • size and from: Control pagination through the indexed result window.

  • _source: Restricts returned source fields, reducing response size and source-fetch overhead.

  • sort: Defines result ordering, optionally using document values or fielddata.

  • aggs: Short for aggregations; computes summaries such as counts, averages, and histograms.

The DSL has leaf queries, which test one condition, and compound queries, which combine conditions. Mapping controls how values are indexed: text fields are analyzed for full-text search, whereas keyword fields preserve exact values.

Core Query Types

Purpose Query Important behavior
Full-text search match Analyzes query text with the field’s search analyzer and matches indexed terms
Exact-value lookup term Does not analyze its input; normally used with keyword, numeric, boolean, or date fields
Phrase matching match_phrase Requires analyzed terms to occur in the specified order using positional information
Boolean logic bool Combines must, should, must_not, and filter clauses
Numeric or date bounds range Supports operators such as gte, gt, lte, and lt
Field presence exists Matches documents containing an indexed value for the field
Vector retrieval knn Finds approximate nearest neighbors, with product-specific syntax and options

Within bool, must and qualifying should clauses affect the relevance score. Clauses in filter context do not score results and are preferable for exact business constraints because eligible filters can be cached. Use minimum_should_match when precise control over optional clauses is required.

Elasticsearch uses BM25 by default for full-text relevance scoring. Term frequency, inverse document frequency, field length, and query structure therefore affect _score.

How Search Executes

  1. A coordinating node receives and parses the request.

  2. Elasticsearch rewrites the query into an internal execution plan.

  3. The request is routed to the relevant index shards.

  4. Each shard executes its portion against its Lucene indexes.

  5. Shard-level hits, scores, and aggregation results are combined.

  6. Selected document data is fetched and returned in the search response.

Lucene segments contain term dictionaries and inverted-index postings used for matching. Stored fields are separate structures used to retrieve selected result fields and are not the matching structures.

Search is near real-time: a successful index request does not guarantee immediate visibility because search visibility depends on the refresh process.

Aggregations and Vector Search

Aggregations summarize the documents selected by the search:

  • terms: Counts values, commonly from a keyword or numeric field.

  • filter: Narrows the aggregation’s input.

  • avg, sum, min, max: Calculate numeric metrics.

  • date_histogram: Groups documents into time intervals.

  • nested: Aggregates documents within nested-object mappings.

Avoid enabling fielddata on analyzed text fields for aggregations. That loads values into heap and can cause severe memory pressure. Prefer a keyword multi-field or an appropriate numeric, date, or keyword mapping.

Current Elasticsearch 8.x releases provide a top-level knn search clause; it is not placed directly inside bool. OpenSearch’s k-NN plugin exposes a knn query with a different request shape and its own filtering, engine, and candidate-selection behavior. Elasticsearch also supports combining vector retrieval with a query filter, but its parameter names and defaults should not be assumed to work unchanged in OpenSearch.

Production Best Practices and Failure Modes

  • Control expensive searches: Leading wildcards, broad regular expressions, expensive scripts, and high-cardinality aggregations can consume substantial CPU and heap.

  • Avoid deep pagination: Large from plus size values increase work and memory. Use search_after, preferably with a point in time, for sequential traversal.

  • Inspect partial failures: Check timed_out, _shards.failed, and _shards.failures. A timed-out request can return partial results, and cancellation of underlying work is not necessarily immediate.

  • Protect mappings: Dynamic mapping can create excessive field counts. Define important mappings explicitly and enforce appropriate mapping limits.

  • Monitor production behavior: Track search latency, rejected tasks, CPU, JVM garbage collection, heap pressure, disk watermarks, unassigned shards, and indexing pressure.

  • Apply least privilege: Use TLS, role-based access control, scoped API keys, and restricted read access for sensitive indices.

  • Size shards pragmatically: A common starting target is approximately 10–50 GB per shard, but validate it against workload, hardware, recovery objectives, and replica placement. Avoid both undersized shards and unnecessarily large shards.

  • Test compatibility separately: Core query syntax is similar between Elasticsearch and OpenSearch, but Elasticsearch-specific APIs, plugins, k-NN behavior, and administrative procedures must not be assumed to work unchanged in OpenSearch.

29

What is the difference between query context and filter context in Elasticsearch?

Query context calculates a relevance score for matching documents, while filter context determines whether documents match without contributing to _score. Elasticsearch uses these contexts to distinguish ranking conditions from reusable inclusion and exclusion constraints.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Difference

Clause or API Context Contributes to _score Typical use
bool.must Query Yes Required matching conditions that should affect ranking
bool.should Query Yes, when matched Optional or required conditions used for ranking
bool.filter Filter No Hard constraints such as status, tenant, language, or price
bool.must_not Filter No Excluding documents that match a condition
post_filter Filter No Filtering returned hits after aggregations are calculated

The context is determined by where a clause appears, not by the query type alone. For example, a match query inside bool.filter is analyzed but does not contribute to the score, while the same query inside bool.must is analyzed and scored.

How It Works

  • Query context: Elasticsearch evaluates the query and calculates a relevance score. With the default BM25 similarity, the score considers term frequency, inverse document frequency, and field-length normalization.

  • Filter context: Elasticsearch performs Boolean matching without calculating a relevance score for that clause. Reusable filter clauses may benefit from Elasticsearch’s segment-level query cache, but caching is an optimization rather than a guarantee and depends on the query structure and cache policy.

  • Score calculation: A query containing only filter clauses returns matching documents with _score: 0.0. When scored and filter clauses are combined, the query-context clauses determine the final relevance score.

  • Analysis behavior: Context does not determine whether a query analyzes its input. A term query is exact and does not analyze its value; a match query analyzes text according to the field’s mapping and analyzer.

When to Use Each Context

Use query context when ranking matters:

  • Full-text relevance searches

  • “Most relevant results first” queries

  • Search suggestions and content ranking

  • Searches where term frequency or field length should influence ordering

Use filter context when only inclusion or exclusion matters:

  • Restricting results to an active or published status

  • Filtering by tenant, region, language, or category

  • Applying date and numeric ranges

  • Excluding deleted, blocked, or discontinued documents

  • Combining hard business constraints with a relevance query

For example, a product search could use a match query in must to rank products by textual relevance, while status, price, and discontinued belong in filter or must_not.

A bool.should clause contributes to the score when it matches. It is optional when the same Boolean query has a matching must or filter clause; without must or filter, at least one should clause must match unless minimum_should_match specifies otherwise.

Production and Operational Considerations

  • Performance: Filters avoid relevance scoring and can reduce repeated work, but they are not automatically fast. Expensive scripts, wildcard patterns, poorly designed mappings, and low-selectivity filters can still consume significant CPU and memory.

  • Caching: Monitor query-cache hit rate, cache size, memory pressure, and segment churn. Do not assume every filter is cached; cacheability depends on its role in the query and Elasticsearch’s cache usage policy.

  • Aggregations: A filter inside the main query affects the documents used to calculate aggregations. post_filter affects returned hits without changing aggregation buckets.

  • Security: Application-level filters can implement document visibility rules, but sensitive tenant isolation should use platform controls such as document-level security where available.

  • OpenSearch: OpenSearch provides the same bool.must, bool.filter, bool.must_not, and post_filter semantics. Cache implementation details and plugin behavior can differ, so performance should be measured independently.

  • Monitoring: Track search latency, slow logs, CPU usage, query-cache statistics, shard count, rejected requests, and filter selectivity when diagnosing production incidents.

30

How do match and term queries differ in Elasticsearch?

A match query analyzes its input before searching, making it suitable for full-text fields such as text. A term query does not analyze its input and therefore performs an exact lookup, making it appropriate for identifiers, codes, statuses, and keyword fields.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

Core Difference

Query Input analysis Typical field Result behavior
match Uses the field’s search analyzer text Finds documents containing analyzed terms
term No analysis keyword, numeric, date, boolean Matches one exact indexed value or token

Both Elasticsearch and OpenSearch use Lucene internally. Their match and term semantics are materially the same; differences are mainly in surrounding platform features, security, and plugins.

For a text field, the original value is generally indexed as multiple analyzed tokens. For example, "Wireless Headphones" might be indexed as wireless and headphones. A match query performs the same analysis on its input and then searches for those tokens.

A keyword field stores the complete value as one term. A term query can therefore match an exact value such as WH-100 without tokenization.

Relevance and Filtering

  • Match query: Usually contributes to relevance scoring. Elasticsearch and OpenSearch use BM25 for full-text scoring, considering term frequency, document frequency, and field length.

  • Term query: Produces an exact match and normally has a constant score, so it is commonly placed inside a bool.filter query.

  • Boolean conditions: term is useful when the value must be exact, while match is useful when users enter natural-language search text.

  • Default matching: match uses OR by default. Set "operator": "and" when all analyzed terms must be present.

Use match_phrase rather than match when word order matters, and use match_phrase_prefix for suitable prefix-style phrase searches.

Production Usage

  • Product IDs and SKUs: Index them as keyword and query them with term.

  • Statuses and categories: Use keyword with term filters when exact values are required.

  • Names and descriptions: Use text with match queries.

  • Case-insensitive exact matching: Configure a lowercase normalizer on the keyword field or normalize values before indexing.

  • Multi-field search: A common mapping uses text with a keyword subfield, supporting both relevance search and exact sorting or filtering.

  • Large datasets: Exact filters are efficient because they avoid query-time tokenization and can use cached filter results.

Common Pitfalls

  • Using term on an analyzed text field is case-sensitive and token-specific. Searching for "Wireless Headphones" will usually not match a field indexed as separate tokens.

  • Using match on a keyword field does not make it a fuzzy or full-text search. With the default keyword analyzer, the entire input is treated as one term and case is preserved.

  • term is not equivalent to a general exact-value match on text; it matches one Lucene term, which may be only one token of the original value.

  • Do not use leading wildcards on large production indices for low-selectivity lookups, because they can create expensive scans and excessive resource usage.

31

How does an Elasticsearch bool query combine must, should, filter, and must_not?

A bool query combines leaf or nested queries into a boolean expression. must and filter are hard requirements, should can add optional relevance signals, and must_not excludes documents.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Clause Semantics

Clause Matching behavior Scoring behavior
must Every clause must match Contributes to _score
should Optional under certain conditions Contributes to _score when matched
filter Every clause must match Does not contribute to _score
must_not Matching documents are excluded Does not contribute to _score

Multiple must or filter clauses are conjunctive: all must match. Multiple should clauses are disjunctive: a document can satisfy any of them. When must or filter is present, should is optional; otherwise, at least one should must match by default. Every must_not clause must fail to match for a document to be returned.

Matching and Scoring Rules

  • Default should behavior: Without must or filter, minimum_should_match defaults to 1; with either of them, it defaults to 0.

  • Explicit control: "minimum_should_match": 1 requires at least one should clause even when must or filter is present. With multiple should clauses, it requires at least one, not all of them.

  • Scoring: must and should participate in relevance scoring through their child queries. Elasticsearch’s default relevance algorithm is BM25, using term frequency, document frequency, and field length.

  • Filtering: filter and must_not constrain matching without contributing to _score. Frequently reused filters may benefit from Elasticsearch’s query cache.

  • Empty bool query: {"bool": {}} matches every document with a score of zero, so it is usually an accidental and expensive way to return the whole index.

  • Nested bool queries: More complex conditions can be represented by placing another bool query inside a clause.

Code Example

First, create an index with suitable field types:

PUT /interview-products
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text"
      },
      "description": {
        "type": "text"
      },
      "status": {
        "type": "keyword"
      },
      "discontinued": {
        "type": "boolean"
      }
    }
  }
}

Then execute the bool query:

POST /interview-products/_search
{
  "_source": ["name", "description", "status"],
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": {
              "query": "wireless headphones",
              "operator": "and"
            }
          }
        }
      ],
      "filter": [
        {
          "term": {
            "status": "active"
          }
        }
      ],
      "should": [
        {
          "match": {
            "description": "noise cancelling"
          }
        }
      ],
      "must_not": [
        {
          "term": {
            "discontinued": true
          }
        }
      ],
      "minimum_should_match": 0
    }
  }
}

This requires both analyzed terms to match in name, requires status to be active, and excludes discontinued products. The description match is optional but increases relevance when it matches.

Distributed and Operational Behavior

  • Each shard evaluates the bool query independently. The coordinating node merges and ranks shard-level results; BM25 statistics are shard-local, so scores can differ between shards.

  • filter and must_not impose non-scoring constraints. Aggregations operate on the documents selected by the complete search query.

  • Replicas provide failover and can serve searches, but consume additional resources. A green cluster has every primary and replica allocated; yellow means some replicas are unallocated; red means at least one primary is unavailable.

  • Start with reasonably sized shards rather than many tiny shards. A shard size in the tens of gigabytes is a common starting point, but ingestion rate, recovery time, hardware, and latency targets should determine the final value.

  • Monitor latency, CPU, heap pressure, garbage collection, disk watermarks, rejected tasks, slow logs, and cluster health. Protect production clusters with TLS, role-based access control, audit logging, and limits on expensive searches. Use snapshots for recovery and index lifecycle management for retention and rollover.

OpenSearch Differences

The bool-query DSL and clause behavior are essentially the same in OpenSearch 2.x/3.x. Both platforms support dense_vector fields and knn queries, but OpenSearch’s approximate vector search is provided through its k-NN plugin, while Elasticsearch implements its vector-search functionality natively. Their security and plugin ecosystems also differ.

Best Practices

  • Use filter for exact categorical constraints such as tenant, region, account type, or publication status.

  • Query keyword fields with term or terms; query analyzed text fields with match.

  • Set minimum_should_match explicitly when the optionality of should matters.

  • Keep scored clauses selective, and avoid very large must_not lists when permitted states can be modeled explicitly.

  • Test expected matching behavior with targeted queries, and use the profile API only when its overhead is acceptable.

32

How do range, prefix, wildcard, and fuzzy queries work in Elasticsearch?

Range, prefix, wildcard, and fuzzy queries match documents using different comparison rules. Range queries select values within boundaries, prefix queries match terms beginning with text, wildcard queries use pattern characters, and fuzzy queries tolerate small spelling differences.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x for the core queries; Elasticsearch 8.7+ and OpenSearch 2.4+ for the `wildcard` field type

Query Semantics

Query Typical field Matching rule Scoring behavior
range Numeric, date, IP, keyword Field value is greater than, less than, or equal to specified bounds Normally used as a filter; matches are not meaningfully ranked
prefix keyword Indexed term starts with the supplied prefix Usually constant-score
wildcard keyword Entire term matches a pattern containing * and ? Usually constant-score
fuzzy keyword Term is within a configurable edit distance of the query value Can participate in relevance scoring
  • Range query: Supports gt, gte, lt, and lte. Numeric and date fields use their native comparison semantics, while keyword fields use lexicographical ordering. Date strings can include date math.

  • Prefix query: Matches SKU-10 and SKU-123 for the prefix SKU-1, but not X-SKU-10. It matches the beginning of a complete indexed term, not an arbitrary substring.

  • Wildcard query: Supports * for zero or more characters and ? for exactly one character. A pattern such as SKU-1*? matches complete terms satisfying that shape.

  • Fuzzy query: Allows insertions, deletions, substitutions, and transpositions. Elasticsearch supports at most two edits. fuzziness defaults to AUTO, which selects zero edits for terms of length 0–2, one edit for length 3–5, and two edits for longer terms. Explicit values are easier to operate predictably.

Analysis and Lucene Behavior

These are primarily term-level queries: they do not perform the same text analysis as a match query. Query them against a keyword field, often using a .keyword multi-field for analyzed text.

For example, a match query against name analyzes both the indexed text and query into tokens. A prefix query against name.keyword instead operates on the entire stored keyword value.

Under the hood, Lucene searches its inverted index, which maps terms to document-frequency postings inside immutable segments:

  • Range queries can use specialized numeric and date indexes.

  • Prefix and wildcard queries often use automaton-style matching over the term dictionary.

  • Fuzzy queries generate candidate term variations, subject to limits such as max_expansions, which defaults to 50.

Prefix and wildcard queries do not calculate meaningful BM25 relevance scores. Put them in a bool.filter clause to make that behavior explicit.

Code Example

These requests are valid Kibana Dev Tools requests:

# Create keyword fields for term-level queries and fields for range queries.
PUT products
{
  "mappings": {
    "properties": {
      "sku": { "type": "keyword" },
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "price": { "type": "double" },
      "created_at": { "type": "date" }
    }
  }
}

# Match products whose price is at least 100 and less than 500.
GET products/_search
{
  "query": {
    "range": {
      "price": {
        "gte": 100,
        "lt": 500
      }
    }
  }
}

# Match indexed SKU terms beginning with SKU-1.
GET products/_search
{
  "query": {
    "prefix": {
      "sku": "SKU-1"
    }
  }
}

# Match complete SKU terms beginning with SKU-1.
GET products/_search
{
  "query": {
    "wildcard": {
      "sku": {
        "value": "SKU-1*"
      }
    }
  }
}

# Match John with one allowed edit from the typo Jhon.
GET products/_search
{
  "query": {
    "fuzzy": {
      "name.keyword": {
        "value": "Jhon",
        "fuzziness": 1,
        "prefix_length": 1,
        "transpositions": true
      }
    }
  }
}

The final request can match a keyword value such as John because Jhon and John differ by a transposition.

Production Failure Modes

Broad patterns and fuzzy searches can expand into many candidate terms, causing:

  • High search latency and CPU consumption

  • Heap pressure and increased garbage collection

  • Rejected search tasks under load

  • Slow or expensive queries across every shard, with one eligible copy queried per shard

Increasing shard count does not inherently make these queries cheaper; it increases overall fan-out. Monitor search latency, query time, search-thread-pool rejections, heap usage, CPU, and segment count. Alert on unusual query patterns as well as aggregate latency.

Leading-wildcard patterns such as *123 are especially expensive because they cannot be resolved as efficiently as a bounded term or prefix. Restrict user-supplied patterns, validate allowed prefixes and maximum pattern length, and bound fuzzy expansion with max_expansions.

Best Practices and OpenSearch Differences

  • Use range inside bool.filter when only membership matters.

  • Use exact keyword fields for prefix, wildcard, and fuzzy matching; use match for analyzed natural-language search.

  • Prefer explicit fuzziness values over AUTO when consistent matching behavior is required.

  • For sustained, pattern-heavy keyword workloads, consider the dedicated wildcard field type: Elasticsearch 8.7+ or OpenSearch 2.4+. Existing keyword mappings require reindexing, and the two implementations are not drop-in compatible.

  • OpenSearch supports the same core query syntax, but field types, plugins, defaults, and the maturity of version-specific features must be checked against the deployed release. Elasticsearch 8.x and OpenSearch both use Lucene, so their core inverted-index and term-expansion behavior is broadly similar.

33

What is a multi_match query in Elasticsearch?

A multi_match query searches for the same text across multiple fields and combines the matching documents into one relevance-ranked result set. It is primarily intended for analyzed, full-text fields and normally uses BM25 scoring in Elasticsearch 8.x and OpenSearch.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x and 3.x

How It Works

A multi_match query is part of the Query DSL and operates in two stages:

  • The query text is analyzed, then its terms are looked up in the inverted indexes of the selected fields.

  • Matches receive relevance scores based on term frequency, inverse document frequency, field length, and field boosts.

With best_fields and most_fields, each field is searched separately using its configured search analyzer. With cross_fields, the fields are treated as one combined field and therefore require compatible analyzers or an explicitly configured shared analyzer. By default, generated match clauses use OR, so at least one analyzed query term must match.

Query Types and Controls

Type Behavior Typical use
best_fields Searches fields independently and uses the highest field score; this is the default Title, description, and name
most_fields Searches fields independently and sums their scores Documents relevant across several fields
cross_fields Treats multiple fields as one combined field Compatible fields such as first and last name
phrase Adds a boosted phrase match plus individual term matches Rewards correctly ordered phrases without requiring one
phrase_prefix Adds a boosted phrase match whose final term is a prefix, plus term matches Search-as-you-type experiences

The operator setting controls the generated match clauses:

  • OR: at least one query term must match; this is the default.

  • AND: every analyzed query term must match. With best_fields or most_fields, the terms must occur in the same field; with cross_fields, they are matched across the combined field group.

Field boosts multiply a field’s score contribution. For example, title^3 applies a boost factor of three to the title field’s contribution.

Code Example

The following request assumes title and description are analyzed text fields. With best_fields and operator: "and", both terms must occur in the same field.

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "wireless headphones",
      "fields": ["title^3", "description"],
      "type": "best_fields",
      "operator": "and"
    }
  }
}

A relevance-scoring multi_match can also be combined with non-scoring filters:

GET products/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "wireless headphones",
            "fields": ["title^3", "description"]
          }
        }
      ],
      "filter": [
        {
          "term": {
            "status": "active"
          }
        },
        {
          "range": {
            "price": {
              "gte": 50,
              "lte": 300
            }
          }
        }
      ]
    }
  }
}

Here, status must be mapped as keyword for exact matching, while price must be a numeric field. The must clause affects the relevance score; filter clauses restrict results without contributing to it.

Production Guidance

  • Choose suitable fields: Use multi_match for analyzed text. For identifiers, status values, and exact codes, prefer term or terms queries on keyword fields.

  • Control phrase behavior: Remember that phrase and phrase_prefix include fallback term matching. Use dedicated match_phrase or match_phrase_prefix queries when phrase relevance must dominate.

  • Use cross_fields carefully: Combine only fields with compatible search analyzers, or specify a shared analyzer.

  • Tune boosts empirically: Validate boosts against representative queries and click or conversion metrics; large boosts can distort ranking.

  • Watch query cost: Broad field lists, many generated terms, phrase queries, and AND conditions can increase CPU use and latency.

  • Monitor production health: Track search latency, timeouts, rejected tasks, JVM heap pressure, CPU, and shard activity. Green means all primaries and replicas are allocated, yellow indicates an unavailable replica, and red indicates an unavailable primary.

  • Plan recovery and upgrades: Configure snapshots and test restores. Use rolling upgrades for compatible node changes, but create a new index and reindex when a mapping change is incompatible.

  • Secure search traffic: Require TLS and authentication, restrict permissions by role, and use field- or document-level security where appropriate.

Elasticsearch and OpenSearch Differences

The core multi_match syntax and scoring behavior are broadly the same in Elasticsearch 8.x and OpenSearch 2.x and 3.x. Version-specific differences are more likely to involve analyzer plugins, synonym handling, security features, and cluster-level capabilities.

multi_match performs lexical retrieval; it is not a vector-similarity query. Elasticsearch and OpenSearch implement embedding search separately through vector fields and features such as knn or script_score.

34

How do phrase queries and slop work in Elasticsearch?

A phrase query matches documents where the analyzed query terms appear in the same relative order and within a configurable positional distance. The slop setting controls how much positional looseness is allowed, while the field’s analyzer determines how the query text and indexed terms are tokenized.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Phrase Queries Work

Elasticsearch stores analyzed terms in an inverted index, along with the positions where each term occurs in a document. A phrase query uses those positions to verify that the terms occur close together and in the expected order.

For example:

Query:      "quick brown fox"
Document:   "The quick brown fox jumped"

The analyzed terms and their positions are approximately:

quick: 1
brown: 2
fox:   3

The query matches because the terms occur consecutively and in order.

The match_phrase query is the usual choice for this behavior. It analyzes the query text using the field’s configured analyzer and creates a phrase query over the analyzed terms.

  • Exact phrase: Requires the terms to occur in the expected positional pattern.

  • Phrase with slop: Allows some positional distance between terms.

  • Analyzer-dependent: Stopwords, synonyms, tokenization rules, and position increments affect what counts as a phrase.

  • Field-dependent: The query should normally target a text field rather than a keyword field. A keyword field is treated as one complete term and is not suitable for ordinary word-level phrase matching.

How Slop Works

slop is an integer that controls positional looseness. It is not a typo tolerance and does not replace fuzzy matching.

  • slop: 0: Requires an exact positional match, subject to analyzer behavior.

  • Positive slop: Allows terms to be separated by intervening terms while preserving the required phrase relationship.

  • Transposed terms: Swapping two adjacent terms generally requires approximately slop: 2 under Lucene’s positional-distance rules.

  • No token substitution: slop does not make "cat" match "hat"; use fuzziness for edit-distance matching when appropriate.

  • Not a proximity filter: A high slop can match many documents, but it does not provide the same relevance as an exact phrase.

For example, with slop: 1, the query "quick brown fox" can match a document containing an extra word between quick and brown, such as:

"The quick red brown fox"

The exact treatment of positions depends on the analyzer and the Lucene implementation, so slop should be tested with the production mapping rather than assumed from the raw query text.

Code Example

The following Kibana Dev Tools request searches for an exact phrase:

GET products/_search
{
  "query": {
    "match_phrase": {
      "description": {
        "query": "wireless noise cancelling headphones",
        "slop": 0
      }
    }
  }
}

This request permits limited positional looseness:

GET products/_search
{
  "query": {
    "match_phrase": {
      "description": {
        "query": "wireless noise cancelling headphones",
        "slop": 2
      }
    }
  }
}

Use the match_phrase_prefix form when the final phrase term should match a prefix:

GET products/_search
{
  "query": {
    "match_phrase_prefix": {
      "description": {
        "query": "wireless noise cancelling head",
        "slop": 0
      }
    }
  }
}

For multi-field phrase searching, use multi_match with "type": "phrase":

GET products/_search
{
  "query": {
    "multi_match": {
      "query": "wireless headphones",
      "type": "phrase",
      "slop": 1,
      "fields": ["description", "name"]
    }
  }
}

match_phrase and multi_match with type: phrase are available in OpenSearch 2.x as well, but parameter support should be checked against the specific OpenSearch release. Do not assume Elasticsearch 8.x-only options exist in OpenSearch.

Query Choices and Relevance

Query Intended use Positional behavior
match General full-text search Matches individual terms without requiring phrase order
match_phrase Exact or near-exact phrase Uses term positions and supports slop
match_phrase_prefix Phrase whose final term is incomplete Matches a prefix for the final term
multi_match (type: phrase) Phrase search across several fields Runs match_phrase on each field and supports slop

Phrase queries generally score differently from ordinary match queries because they consider phrase frequency and positional information. An exact phrase usually has stronger relevance than a loose phrase, although the final score still depends on BM25 term statistics, boosts, document length, and query construction.

Production Best Practices

  • Use slop: 0 for strict phrase requirements; increase it only when near-order matches are acceptable.

  • Validate slop behavior against the production analyzer, especially when fields use synonyms, stopwords, shingles, or position increments.

  • Avoid very large slop values. They broaden the candidate set, can reduce precision, and may increase query cost.

  • Do not use slop for typo tolerance; use a carefully bounded fuzziness option where edit-distance matching is intended.

  • Monitor latency, result counts, and relevant-query metrics such as query_latency, fetch_latency, and rejected searches.

  • Test query behavior after mappings, analyzers, or synonym configuration changes. Phrase semantics can change even when the JSON query remains unchanged.

  • Keep phrase queries on appropriately analyzed text fields and use keyword fields for exact filters, aggregations, and strict term equality.

35

How does BM25 relevance scoring work in Elasticsearch?

BM25 is Elasticsearch’s default relevance model for analyzed text fields. It combines term frequency, inverse document frequency, and field-length normalization, with term-frequency saturation so repeated matches help without dominating the score.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.12–3.x

How the BM25 Score Is Calculated

For each analyzed query term, BM25 calculates a contribution based on three factors:

  • Term frequency (f): The number of times the term occurs in the document field. Additional occurrences increase the score, but with diminishing returns.

  • Inverse document frequency (IDF): A measure of how distinctive the term is. Lucene uses ln(1 + (N - n + 0.5) / (n + 0.5)), where N is the number of documents in the field’s collection and n is the number containing the term.

  • Field length (dl): The number of analyzed tokens in the field. Shorter fields are favored because a matching term represents a larger portion of their content. The normalization denominator compares dl with avgdl, the average field length.

The per-term, per-field contribution is proportional to IDF × (f × (k1 + 1)) / (f + k1 × (1 - b + b × dl / avgdl)). Elasticsearch sums the contributions from matching terms and fields to produce _score; boosts can multiply the relevant contributions.

The default parameters are:

  • k1 = 1.2: Controls term-frequency saturation. A higher value allows repeated occurrences to contribute more, while a lower value causes saturation sooner.

  • b = 0.75: Controls field-length normalization. At 0, field length is ignored; at 1, full normalization is applied.

Fields are scored independently, so a term matching a title, description, and tag produces separate field contributions that can be combined.

Analyzer and Query Effects

BM25 operates on tokens produced by the configured analyzers:

  • The document analyzer processes indexed field content.

  • The query analyzer normally processes searches using match, match_phrase, and multi_match.

  • A term query is not analyzed and is normally used for exact keyword, identifier, or enum matching.

  • Queries in bool.filter and bool.must_not do not contribute to _score.

  • bool.should clauses contribute to scoring when the Boolean query has no must or filter clause. When a must or filter is present, should clauses are optional and normally do not affect the score unless minimum_should_match makes them required.

  • Stop words, stemming, synonyms, and tokenization can materially change which terms are scored and therefore affect ranking.

Interpreting Scores

BM25 scores indicate relative lexical relevance rather than probabilities:

  • A higher _score means a stronger match under the current analyzers, similarity settings, and boosts.

  • Scores should generally be compared within the same query and index, not treated as absolute relevance values.

  • Raw scores from unrelated fields, queries, or indexes are not directly comparable.

  • BM25 measures lexical matching, not semantic meaning. Vector or hybrid retrieval is more appropriate when users may express the same concept with different words.

The _score and explain output can be inspected to identify which terms and fields contributed to a result’s ranking.

Production Considerations

  • Shard-level statistics matter: Searches are executed independently on shards, and local document-frequency and field-length statistics contribute to scoring. Indexes with different shard layouts or document distributions can therefore produce small ranking differences, even though identical documents and analyzers are used.

  • Shard sizing remains important: Oversharding increases heap pressure, scheduling work, and operational complexity. Keep primary shards reasonably sized—often in the tens-of-gigabytes range when practical—and validate the choice against indexing throughput, retention, and recovery objectives.

  • Field boosts should reflect intent: A title or product-name match may be more valuable than a body match, but boosts should be validated with representative evaluation queries rather than chosen arbitrarily.

  • Tune conservatively: k1 and b are useful baselines. Change them only through an evaluation set containing realistic queries, relevance judgments, and production-like data.

  • Diagnose regressions systematically: Compare mappings, analyzer output, similarity settings, explain results, deleted-document impact, and shard-local statistics. A shard relocation alone should not change the score.

  • Control query cost: Use role-based access control, request-size limits, timeouts, pagination, and result-window or aggregation limits to prevent expensive searches from consuming excessive CPU and heap.

  • Account for OpenSearch differences: OpenSearch 2.12 and later use BM25 by default for newly created text fields when no explicit similarity is configured. Pre-2.12 indices may retain the earlier TF-IDF-style DefaultSimilarity, so existing mappings should be checked directly.

36

How do you boost Elasticsearch fields and documents?

Relevance boosting increases the _score of matching fields or documents without changing the indexed data. Elasticsearch and OpenSearch provide query-level boost, function_score for per-document signals, and rescore for reranking top search results.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Boosting Works

  • BM25 relevance: Analyzed match and multi_match queries calculate _score using term frequency, inverse document frequency, and field-length normalization.

  • Query boost: The boost parameter is a non-negative floating-point multiplier, normally 1.0. A boost of 4 multiplies that query clause’s score by four; it does not necessarily multiply the final document score.

  • Field boosts: In multi_match, syntax such as title^4 increases the contribution of matches in title.

  • Clause boosts: In a bool query, individual should clauses can have different boosts. filter clauses do not contribute to _score.

  • Query type matters: match analyzes its input, while term performs an exact lookup. A boost cannot make a nonmatching term match.

Choosing a Boosting Method

Goal Mechanism Key consideration
Prioritize a field multi_match with field boosts Use measured weights such as title^3
Prioritize an exact phrase match_phrase with boost Can be combined with other should clauses
Promote specific documents function_score with field_value_factor Derives the score from a numeric document field
Apply a learned relevance signal rank_feature query Intended for rank_feature or rank_features mappings
Rerank initial results rescore query Reorders the top window_size hits per shard
Filter without affecting relevance bool.filter Filtering is independent of scoring

function_score has two important controls:

  • score_mode: Combines function scores; common values include sum, multiply, max, and avg.

  • boost_mode: Combines the function result with the original query score. multiply multiplies the original score by the function result, while sum adds the function result to it.

Production and Operational Impact

  • No reindex required: Query-time boosts do not alter mappings or stored documents. Changing an analyzer only affects newly indexed documents; changing incompatible field data generally requires reindexing.

  • Shard-local statistics: BM25 statistics are calculated per shard, so absolute scores can differ between shards. Rankings should be tested statistically rather than relying on a fixed _score threshold.

  • Replica behavior: Replicas provide availability, not additional scoring capacity. A query normally searches one active copy of each shard.

  • Rescore trade-off: rescore improves final ordering but increases CPU and latency because it evaluates another query over the candidate window.

  • Shard sizing: Start around 20–40 GiB per primary shard and benchmark against the workload. Avoid creating many tiny shards; use replicas primarily for availability and throughput resilience.

  • Cluster health: A yellow cluster usually indicates an unavailable replica, while red indicates a missing primary. Monitor disk watermarks, rejected writes, search latency, CPU, JVM heap pressure, and shard growth.

  • Ingestion and lifecycle: Boosting does not correct bad input normalization; handle that in ingest pipelines. ILM controls index rollover and retention, not query relevance.

  • Promotion signals: Validate popularity_score as non-negative during ingestion. The log1p modifier is undefined at or below -1, and function scores must not be negative.

  • Aggregations and vectors: Bucket counts are unaffected by relevance boosts; a top_hits sub-aggregation can use the resulting _score when ordering its hits. BM25 field boosts do not affect kNN similarity, but hybrid search can apply them to lexical retrieval while kNN ranks vector candidates and the two lists are normalized or reranked. Elasticsearch and OpenSearch expose different hybrid-search and reranking APIs.

  • Reliability and security: Protect boost configuration from untrusted users, enforce TLS and role-based access, and include indices in regular snapshots.

Code Example

Create an index containing text fields and a numeric document-promotion signal:

PUT /products
{
  "mappings": {
    "properties": {
      "title": { "type": "text" },
      "brand": { "type": "text" },
      "description": { "type": "text" },
      "popularity_score": { "type": "float" }
    }
  }
}

Prioritize title and brand matches:

POST /products/_search
{
  "query": {
    "multi_match": {
      "query": "wireless headphones",
      "fields": ["title^4", "brand^2", "description"],
      "type": "best_fields"
    }
  },
  "explain": true
}

Add a document-level promotion score:

POST /products/_search
{
  "query": {
    "function_score": {
      "query": {
        "match": {
          "description": {
            "query": "noise cancelling",
            "boost": 1.0
          }
        }
      },
      "field_value_factor": {
        "field": "popularity_score",
        "factor": 0.2,
        "modifier": "log1p",
        "missing": 0
      },
      "boost_mode": "sum",
      "score_mode": "sum"
    }
  }
}

Here, log1p prevents large popularity values from dominating, and boost_mode: "sum" adds the promotion signal to the normal relevance score.

Best Practices and OpenSearch Differences

  • Calibrate boosts using a representative query set and human relevance judgments; avoid arbitrary large values.

  • Use rank_feature for positive business or relevance signals and function_score for simpler numeric adjustments.

  • Use explain during development to inspect score components, but remove it from routine production searches because it adds overhead.

  • The principal multi_match, function_score, field_value_factor, and rank_feature behavior is similar in OpenSearch, although explanation output and managed-service defaults can differ.

  • Do not depend on the legacy _boost index field. Elasticsearch 8.x does not support it, and OpenSearch’s deprecated compatibility behavior is not portable.

37

How does function_score modify Elasticsearch search scores?

function_score is an Elasticsearch query that wraps another query and modifies its relevance score using signals such as recency, popularity, location, or a business-specific filter. It is useful when document text relevance alone does not represent the ranking required by the application.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How It Works

The wrapped query first calculates its normal relevance score, usually using BM25. One or more functions then calculate additional scores for matching documents. The result combines:

  • Query score: The original text-relevance score.

  • Function score: The combined score from the configured functions.

  • Boost mode: Controls how the query and function scores are combined.

A function without a filter applies to every document returned by the wrapped query. A function with a filter applies only to matching documents.

Common Functions

  • filter and weight: Adds a fixed score to selected documents. For example, in-stock products can receive a boost.

  • field_value_factor: Uses a numeric document field, such as popularity, sales volume, or rating.

  • gauss, linear, and exp decay: Scores documents according to distance from an origin. Common uses include geographic proximity and freshness.

  • random_score: Produces randomized scores, commonly for exploration or diversified result ordering.

  • script_score: Executes a script to calculate a custom score. It is flexible but can increase CPU usage and introduce security risks.

The core query syntax and these functions are supported by both Elasticsearch and OpenSearch, although plugin-provided capabilities can differ.

Score Controls

Two controls combine function and query scores, while max_boost limits the resulting score:

Setting Purpose Values and behavior
score_mode Combines individual function scores before they interact with the query score multiply (default), sum, avg, first, max, min
boost_mode Combines the query score with the combined function score multiply (default), replace, sum, avg, max, min
max_boost Caps the maximum score the query can produce Numeric value; effectively unlimited by default

With score_mode: "sum", the function scores are added. With boost_mode: "multiply", the combined function score is then multiplied by the original query score. With boost_mode: "replace", the function score becomes the final relevance score and the original query score is discarded.

Code Example

The following Kibana Dev Tools request assumes a products index has a numeric popularity field and a boolean in_stock field:

GET products/_search
{
  "query": {
    "function_score": {
      "query": {
        "multi_match": {
          "query": "wireless headphones",
          "fields": [
            "name^2",
            "description"
          ]
        }
      },
      "functions": [
        {
          "filter": {
            "term": {
              "in_stock": true
            }
          },
          "weight": 2
        },
        {
          "field_value_factor": {
            "field": "popularity",
            "factor": 0.5,
            "modifier": "sqrt",
            "missing": 1
          }
        }
      ],
      "score_mode": "sum",
      "boost_mode": "multiply",
      "max_boost": 10
    }
  }
}

Here, relevant in-stock products receive a fixed boost, while popular products receive an additional score based on popularity. The summed function score is multiplied by the original BM25 score and capped by max_boost.

Operational and Security Considerations

  • Validate ranking behavior: Use _explain during development to inspect the query score, individual function contributions, and final score.

  • Avoid excessive boosting: Large weights or boost_mode: "replace" can cause business signals to overwhelm textual relevance.

  • Prefer built-in functions: script_score is computationally expensive at scale. Restrict scripts, avoid expensive operations, and never accept unrestricted user-provided scripts.

  • Account for shard execution: Functions operate on documents within each shard and are not a substitute for globally normalized ranking signals.

  • Monitor latency and saturation: Track search latency, CPU usage, rejected tasks, and query timeouts after introducing complex functions.

  • Use fresh signals carefully: Recency and inventory data may become stale. Update them reliably and define how missing values affect ranking.

  • Apply authorization first: Do not expose ranking boosts for confidential attributes, restricted inventory, or internal business metrics to unauthorized users.

38

How do you debug relevance with the Elasticsearch explain API?

The Elasticsearch explain API computes a relevance-score explanation for one document and one query, showing why that document matched or failed to match. It uses the same query execution and scoring machinery as search while exposing contributions such as term frequency, inverse document frequency, boosts, and individual query-clause scores.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Explain Works

POST /<index>/_explain/<document-id> executes a query for one document and returns:

  • matched: whether the document satisfied the query.

  • explanation.value: the computed relevance score.

  • explanation.description and explanation.details: a tree describing how the score was assembled.

The endpoint runs against the shard containing the document. Reproduce the production query exactly, including query clauses, boosts, analyzers, routing, and any preference used by the application. Scores and descriptions depend on the index contents, similarity settings, document frequencies, and segment state.

Debugging Workflow

  1. Check the mapping. Confirm whether the field is text, keyword, or another type. A keyword field is not analyzed like a text field.

  2. Inspect analysis. Use _analyze with the field to see the tokens produced by its index-time analyzer. Account for lowercasing, stemming, synonyms, stop words, and character filters.

  3. Distinguish query types. A match query analyzes its input, while a term query requires the exact indexed term. For example, term against name.keyword matches "Wireless Headphones" as one keyword value.

  4. Read the score tree. With BM25 similarity, repeated terms and shorter fields usually increase the score, while common terms contribute less because of inverse document frequency. A larger field boost increases that field’s contribution.

  5. Understand bool clauses. must and should clauses can contribute to the score; filter clauses affect membership but not relevance scoring. A constant_score query deliberately replaces normal relevance scoring with a fixed score.

  6. Compare with search. Search with "explain": true to see the explanation attached to actual hits. This is expensive, so enable it selectively rather than for every production search.

Production and Version Considerations

A score from _explain can differ from a distributed search result because the requests select different shards or replicas, use different segment states, or because the search used dfs_query_then_fetch, which computes global document frequencies.

If a newly indexed document is not yet visible, refresh the affected index before testing:

# Make recently indexed documents visible to search and explain.
POST /products/_refresh

Use the same routing value as the application; otherwise the explain request may not locate a custom-routed document. Monitor search latency, error rate, shard failures, heap, disk, and relevance changes. Explain is a diagnostic tool, not a general relevance-monitoring mechanism.

OpenSearch supports the same basic _explain endpoint and Lucene-style scoring model, but exact response descriptions and plugin-specific behavior can differ from Elasticsearch. Validate them against the deployed OpenSearch version.

The endpoint requires appropriate read permissions and can return _source or stored fields. Apply the same authorization and audit controls used for search, especially because document content may be exposed.

Code Example

Explain a document with a production-style match query:

# Explain why this document matched the query.
POST /products/_explain/prod-1
{
  "query": {
    "match": {
      "name": "wireless headphones"
    }
  }
}

Inspect the tokens generated for the mapped field:

# Show the analyzer output for the mapped field.
POST /products/_analyze
{
  "field": "name",
  "text": "Wireless Headphones"
}

Compare the explanation with actual search hits:

# Attach an explanation to each returned hit.
GET /products/_search
{
  "query": {
    "match": {
      "name": "wireless headphones"
    }
  },
  "explain": true
}

Check mapping and shard health when results change unexpectedly:

# Inspect the current field mapping.
GET /products/_mapping

# Inspect shard allocation and health.
GET /products/_cat/shards?v
GET /_cluster/health/products

Best Practices and Common Pitfalls

  • Compare the exact production query rather than a simplified equivalent.

  • Use _analyze before investigating score formulas.

  • Use term only when exact keyword semantics are intended.

  • Preserve the production routing and shard-selection behavior.

  • Run explain for individual documents, not broad result sets.

  • Treat score differences across replicas or times as evidence of index-state changes until proven otherwise.

39

What are aggregations in Elasticsearch?

Aggregations are Elasticsearch queries that compute summary statistics and groupings over matched documents, similar to SQL GROUP BY plus aggregate functions, but executed directly on the inverted index and doc values.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How They Work

Aggregations run in two phases. First, each shard computes partial results locally; then the coordinating node reduces them into the final response. They operate on doc values (column-oriented storage) rather than the inverted index, which makes grouping and numeric summaries efficient. Setting size: 0 skips returning hits and returns only aggregation results, which is the common pattern for dashboards and analytics.

Aggregation Types

Type Purpose Common Examples
Metric Compute a single value or set of values avg, sum, min, max, cardinality, stats, percentiles
Bucket Group documents into buckets terms, range, date_histogram, histogram, filters
Pipeline Operate on the output of other aggregations avg_bucket, derivative, cumulative_sum, moving_avg
Matrix Compute statistics across multiple fields matrix_stats

Key Points

  • terms aggregations need a field with doc values, such as keyword, numeric, date, or IP fields; analyzed text fields are not aggregatable unless fielddata is explicitly enabled, which is usually discouraged.

  • High-cardinality terms aggregations can be memory-heavy; use cardinality for approximate distinct counts instead of very large term buckets.

  • date_histogram is the standard tool for time-series charts; combine it with a metric sub-aggregation.

  • Aggregations respect the query context: only documents matching the query are aggregated.

  • A typical analytics request uses size: 0, a query to limit the document set, and nested aggs for grouping plus metrics such as sum and avg.

Best Practices and Pitfalls

  • Always set size: 0 when you only need aggregations to reduce network and serialization overhead.

  • Use keyword sub-fields for terms aggregations; avoid aggregating on analyzed text.

  • Watch shard-level memory: deep terms with large size can trigger circuit breakers; prefer composite aggregations for paginated large result sets.

  • For exact distinct counts at scale, consider pre-aggregating at ingest time or using cardinality with an acceptable error margin.

  • In OpenSearch, aggregation syntax is largely compatible, but some pipeline and matrix aggregations differ; verify against the OpenSearch documentation for your version.

40

How do bucket, metric and pipeline aggregations differ in Elasticsearch?

Bucket aggregations group documents into sets, metric aggregations calculate values, and pipeline aggregations calculate or transform values using the buckets produced by another aggregation. Elasticsearch and OpenSearch share this core model, although their catalogs of optional aggregations evolve independently.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 3.x core search APIs

Core Differences

Aggregation type Input Output Common examples
Bucket Documents Groups of documents, optionally containing nested metrics terms, date_histogram, range, filter
Metric Documents or values from a bucket One calculated value per aggregation context, such as per bucket avg, sum, min, max, cardinality, percentiles
Pipeline Metrics or buckets from another aggregation Another metric, transformed value, or reordered bucket collection avg_bucket, sum_bucket, derivative, bucket_sort

A bucket aggregation creates a hierarchy. For example, a terms aggregation might group orders by region, with an avg metric nested inside each region. The nested avg remains a metric aggregation because it produces one value for each region bucket.

A pipeline aggregation operates at the bucket level rather than independently scanning documents. It can calculate the average revenue per monthly bucket, derive month-over-month changes, or sort the resulting buckets.

How Execution Works

  • The coordinating node sends the search request to relevant shards.

  • For shardable bucket and metric aggregations, each shard computes partial results.

  • The coordinating node merges those results into the final response.

  • Pipeline aggregations consume the assembled aggregation hierarchy, so they depend on their parent or sibling aggregation’s structure and paths.

  • Bucket aggregations commonly operate efficiently over doc_values, especially for keyword, numeric and date fields.

  • Metric aggregations also commonly use doc_values; some metrics, such as cardinality and percentiles, use approximate data structures.

Aggregations run in the search layer and do not modify stored documents. They still consume CPU, heap memory and I/O, particularly when processing many shards or large numbers of buckets.

Example Use Cases

  • Terms bucket: Count orders per product category or group logs by HTTP status.

  • Date histogram bucket: Produce monthly request counts for a dashboard.

  • Range bucket: Calculate error rates for different latency ranges.

  • Metric aggregation: Calculate average order value, total revenue, approximate unique-user counts, or response-time percentiles.

  • Pipeline aggregation: Compare revenue between months, calculate a cumulative total, or rank categories by their nested metric.

For searchable text, the text field itself is generally not aggregatable. A common mapping pattern exposes a keyword multi-field and runs the bucket aggregation against that subfield.

Accuracy and Performance

  • terms counts can be approximate in a distributed cluster because each shard initially selects candidate terms. Increasing shard_size improves the candidate pool, while collect_segments can improve count accuracy at additional cost.

  • cardinality is approximate and uses HyperLogLog++ rather than maintaining an exact set of every distinct value.

  • percentiles is approximate and commonly uses a t-digest, trading exactness for lower memory usage.

  • High-cardinality buckets can create large responses and consume substantial coordinating-node memory.

  • index.search.max_buckets is an important per-index guardrail. Raising it permits larger aggregations but increases memory pressure and result-merging work.

  • Timeouts can produce partial results with timed_out: true; clients should not automatically interpret such responses as complete.

Shard sizing remains workload-dependent. For search-heavy indices, many teams begin with shards around 10–30 GB and then benchmark using realistic mappings, aggregations and indexing rates. Oversized shards slow recovery and merging, while excessive shard counts increase cluster metadata and coordination overhead.

Elasticsearch and OpenSearch Differences

The basic terms, date_histogram, range, filter and metric syntax is broadly compatible between Elasticsearch 8.x and OpenSearch 3.x.

The core pipeline taxonomy also overlaps substantially. OpenSearch 3.x supports common bucket-to-bucket metrics such as avg_bucket, sum_bucket, min_bucket, max_bucket, stats_bucket and percentiles_bucket. It also supports pipeline operations including bucket_sort, bucket_selector, cumulative_sum, cumulative_avg, derivative and moving_fn; these are not Elasticsearch-only.

Elasticsearch and OpenSearch release optional or less common aggregations independently, so exact feature availability should be checked against the product and minor version being deployed rather than inferred from the three aggregation categories.

Production and Security Guidance

  • Monitor aggregation latency, search thread-pool rejections, CPU, garbage collection, heap usage and circuit-breaker activity.

  • Avoid unrestricted high-cardinality aggregations over unbounded fields such as request IDs, user IDs or raw URLs.

  • Use size to control returned buckets and shard_size to balance accuracy against distributed execution cost.

  • Set appropriate timeouts and handle partial responses.

  • Protect sensitive indices with least-privilege roles, document-level or field-level controls, TLS and audit logging where supported. Aggregation responses can reveal values even when individual documents are not displayed.

  • Test aggregation behavior during index rollover and shard reallocation. A healthy cluster can still experience high aggregation latency because aggregation work competes with indexing and other searches.

41

Why are Elasticsearch terms aggregation counts sometimes approximate?

A terms aggregation groups matching documents into buckets by field value. Its per-bucket doc_count can be approximate because each shard ranks only a limited set of terms; the coordinating node merges those shard-level candidates and cannot recover counts discarded before the merge.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How It Works

A terms aggregation is normally used with a keyword field:

  • Each distinct value forms a bucket.

  • doc_count represents the number of matching documents containing that value.

  • By default, buckets are ordered by document count and the ten highest-count buckets are returned.

  • A document can contribute to multiple buckets when a keyword field contains multiple values, so bucket counts need not sum to the total number of documents.

  • The aggregation includes only documents matched by the request’s query or filter.

Each shard evaluates the aggregation independently. Elasticsearch uses Lucene’s term dictionaries and global ordinals to resolve term values and calculate shard-local counts. The coordinating node then combines the candidates returned by the shards into the final response.

Why Counts Can Be Approximate

The approximation results from distributed execution:

  1. Each shard selects a limited number of candidate terms using shard_size.

  2. A globally common term might rank below other terms on a particular shard because its distribution is skewed.

  3. If that shard omits the term from its candidate set, its document count is missing from the final bucket.

  4. The coordinating node sums the counts returned by the shards, potentially undercounting the global count.

The response exposes useful diagnostics:

  • doc_count_error_upper_bound: The maximum possible undercount for a returned bucket.

  • doc_count_error_lower_bound: The lower bound, returned when show_term_doc_count_error is enabled.

  • sum_other_doc_count: An approximate sum of counts associated with omitted terms; it does not restore counts already lost during shard-level candidate selection.

This is not simple rounding. The coordinator never receives the complete term counts for all terms on every shard. If a term is returned by every relevant shard, its bucket count is exact relative to the search-visible documents.

Improving Accuracy and Exact Enumeration

The size option controls the final number of buckets, while shard_size controls how many candidates each shard may contribute. The default shard_size is derived from size, but it remains a candidate-selection heuristic. Increasing it generally improves accuracy when terms are distributed unevenly across shards, at the cost of additional memory, network traffic, and coordination work.

Enabling show_term_doc_count_error exposes both error bounds for returned buckets. These bounds help identify when a result is unreliable, but they do not make the aggregation exact.

For exhaustive traversal of high-cardinality fields, use a composite aggregation with a terms source. Composite aggregations paginate through unique keys using the response’s after_key, supplied as the request’s after value. Each returned bucket has an exact doc_count for the search-visible documents associated with that key. They avoid terms aggregation’s per-shard top-N truncation, but they are generally more expensive.

Production and OpenSearch Considerations

Concern Guidance
High cardinality Avoid unrestricted terms aggregations on IDs or unbounded event fields; use composite aggregation for exhaustive results.
Skew Increase shard_size and inspect the reported error bounds.
Resource usage Terms aggregations fan out to every shard and can create significant heap and network pressure.
Monitoring Track search latency, rejected tasks, JVM heap, shard count, and relocation activity using /_nodes/stats/jvm,search, /_cat/shards?v, and /_cluster/health.
Security Grant users search access only to the indices and fields required by the aggregation.
Data changes Counts reflect the search-visible index state at query time, not a transactionally consistent snapshot across shards.

OpenSearch 2.x and 3.x provide the same terms, shard_size, error-bound, and composite aggregation semantics. Operational tooling differs: OpenSearch normally uses OpenSearch Dashboards and its plugin ecosystem, while Elasticsearch integrates with the Elastic Stack. Accuracy and performance still depend on shard count, cardinality, data skew, and available heap.

42

What are date histogram aggregations in Elasticsearch?

A date histogram aggregation groups documents into buckets based on a date or timestamp field, such as errors per hour or requests per day. Each bucket reports its doc_count and can contain nested metric aggregations for values such as average latency or total bytes.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How It Works

The aggregation evaluates matching documents and assigns each document to a time bucket:

  • The target field must be mapped as date or date_nanos.

  • Documents outside the search time range are excluded.

  • Buckets are returned in ascending chronological order.

  • A doc_count represents the number of documents in that bucket.

  • Nested aggregations can calculate metrics within each bucket, such as avg, sum, min, or max.

In a distributed cluster, each shard computes local buckets. The coordinating node then merges them into the final result. Because the aggregation runs across all matching shards, expensive ranges, very small intervals, or excessive bucket counts can increase CPU usage, heap pressure, and search latency.

Date histogram aggregations are also usable in Kibana Lens time-series visualizations and Elasticsearch SQL date grouping functions.

Interval and Bucket Options

Option Purpose
calendar_interval Calendar-aware interval such as 1d, 1w, 1M, 1q, or 1y. It follows calendar boundaries and timezone offset changes.
fixed_interval Fixed-duration interval such as 5m, 1h, or 1d. A fixed day is always 24 hours, even across daylight-saving transitions.
time_zone Determines where calendar-based bucket boundaries begin. Use an IANA timezone such as America/New_York or UTC.
offset Shifts bucket boundaries, useful when reporting by business hours.
min_doc_count Controls whether buckets below a document threshold are returned.
extended_bounds Forces the response to cover a specified minimum and maximum time, including otherwise empty intervals.
missing Assigns documents with a missing date value to a specified bucket.
format Formats the returned key_as_string, commonly using a Java date-time pattern.
order Changes bucket ordering, although chronological ordering is normally most useful.

Use calendar_interval for reports based on months, quarters, years, or local calendar days. Use fixed_interval for predictable durations such as five-minute monitoring windows. Mixing the two can create confusing boundaries around daylight-saving changes.

A typical request uses size: 0, a range query on the date field, and an aggs object containing date_histogram with field, fixed_interval, time_zone, and optional bounds. The range filter limits the documents processed by the aggregation; filtering with post_filter instead would apply the filter after aggregations and would not reduce aggregation work.

Production Considerations

  • Query scope: Restrict the time range and relevant services. Daily indices with rollover and index lifecycle management are often easier to query than one enormous index.

  • Bucket count: Avoid intervals that create thousands of buckets per shard. extended_bounds can also generate empty buckets, so use it deliberately.

  • Shard sizing: A common starting point is roughly 10–50 GB per primary shard, adjusted for throughput, retention, and recovery objectives. Too many small shards increase cluster coordination overhead.

  • Cluster health: A green cluster has all primaries and replicas assigned. A yellow cluster usually still returns complete aggregation results but lacks replica redundancy. A red cluster has an unavailable primary and may return incomplete or missing data.

  • Visibility: Newly indexed documents become searchable after a refresh. Long-running reports may need a point-in-time view for stable results.

  • Monitoring: Track aggregation latency, CPU, heap pressure, rejected tasks, shard count, disk watermarks, and bucket cardinality.

  • Recovery: Use snapshots and regularly test restores. Derived aggregations can be rebuilt from the underlying source data.

Elasticsearch and OpenSearch Differences

The core date_histogram API, including calendar_interval, fixed_interval, time_zone, and extended_bounds, is substantially the same in Elasticsearch 8.x and OpenSearch.

Operationally, self-managed Elasticsearch 8.x enables its security layer by default, whereas OpenSearch security depends on the installed distribution and security-plugin configuration. In both systems, expose search endpoints only through authenticated and encrypted access, restrict aggregation permissions with role-based access control, and enable appropriate audit logging.

43

How does Elasticsearch paginate results, and why is deep pagination costly?

Pagination in Elasticsearch is normally implemented with from/size, search_after, or scroll. Deep pagination is costly because every shard must evaluate matching documents and determine an increasingly large top-results set before the coordinating node merges the final page.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Pagination Mechanisms

  • from/size: from is a zero-based document offset, and size is the number of results requested. This is simple and supports random page access, but requesting a large offset causes the cluster to process and merge all preceding candidates.

  • search_after: The caller supplies the complete sort array from the previous result as the cursor for the next request. It avoids calculating a large offset and is suitable for forward-only traversal. Without a point in time, concurrent indexing, updates, and deletions can cause documents to be missed or returned more than once.

  • Point in time (PIT) with search_after: A PIT provides a consistent view of the selected shards while the traversal continues. Use the same query and deterministic sort on every request, pass the latest returned PIT ID, and copy the final hit’s complete sort array into search_after.

  • scroll: Scroll maintains a server-side search context for sequential processing, such as exports or reindexing. It consumes cluster resources and is not intended for interactive pagination.

Why Deep Pagination Is Expensive

A search request is distributed across the relevant shards:

  1. The coordinating node sends the request to each shard.

  2. Every shard evaluates matching documents and determines its top from + size candidates.

  3. The coordinating node merges those shard-level results and ranks the global top results.

For from: 10000 and size: 20, each shard must determine its top 10,020 candidates. This increases sorting and selection work, JVM heap pressure, CPU consumption, and network traffic. The default index.max_result_window is 10,000, so from + size cannot exceed that limit. Increasing the limit permits deeper requests but does not remove their underlying cost.

Choosing a Pagination Method

Method Cross-request consistency Resource profile Appropriate use
from/size No guarantee between separate requests Cost grows with from Shallow, random-access pagination
search_after without PIT Writes can cause omissions or duplicates Lower cost than large offsets Forward-only traversal with explicit tie-breakers
PIT + search_after Consistent view of the PIT Efficient for deep traversal User-facing cursors and large exports
scroll No strict snapshot guarantee across pages Holds a server-side search context Sequential batch processing

Production Guidance and OpenSearch Differences

  • Use deterministic sorting: Prefer keyword, numeric, date, or other doc-value-backed fields. Analyzed text fields are unsuitable as deterministic sort fields. Without a PIT, include a unique, stable tie-breaker; with a PIT, _shard_doc is supported as the final tie-breaker.

  • Manage PIT lifetime: Use a reasonable keep_alive, continue using the latest PIT ID returned by each request, and close the PIT when traversal ends. Long-lived PITs can retain search contexts and old Lucene segments.

  • Prefer cursors for large result sets: Cursor pagination does not naturally support numbered pages or an exact total count. If those are hard requirements, explicitly materialize stable page identifiers and totals in a dedicated store or index, accepting the associated staleness and maintenance costs.

  • Monitor production pressure: Track search latency, JVM heap, CPU, rejected search tasks, shard fan-out, and slow queries. Deep pagination can expose excessive per-shard work, but it does not by itself prove that shards are incorrectly sized.

  • Handle node and cluster failures: PIT and scroll state are server-side search context, not durable job checkpoints. After severe node or cluster failures, restart the traversal with a new PIT and deduplicate downstream results if necessary.

  • Secure access: Require authenticated, encrypted connections and grant only the index-search and PIT permissions needed by the application.

  • Vendor differences: Elasticsearch 8.x and OpenSearch 2.x/3.x support the same core from/size, PIT, search_after, and scroll concepts. Security defaults, distribution packaging, and cluster-management tooling differ; Elasticsearch 8.x enables its security features by default, whereas OpenSearch deployments commonly configure or enable the security plugin explicitly.

44

How do from/size, search_after and scroll compare in Elasticsearch?

from/size, search_after, and scroll are Elasticsearch pagination strategies with different performance, consistency, and operational characteristics. from/size is appropriate for shallow pages, search_after is preferable for deep sequential pagination, and scroll is intended primarily for bulk extraction.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.4+ (OpenSearch PIT support begins in 2.4).

How They Work

  • from/size: Offset-based pagination. The coordinating node requests enough sorted results from each shard to satisfy from + size, merges the shard results, and trims them to the requested window. Cost increases with depth because deeper pages require larger per-shard candidate sets.

  • search_after: Cursor-based pagination. The client sends the complete sort tuple from the last hit, and Elasticsearch returns matching documents after that tuple. It avoids the offset penalty but requires a deterministic sort with a unique tie-breaker.

  • scroll: Server-side extraction. The first request creates a search context and returns a scroll_id; subsequent requests use that ID to retrieve the next batch. Scroll preserves search state and cursor position, not point-in-time snapshot isolation.

Comparison

Mechanism Best fit Deep pagination Consistency under writes Main limits
from/size First few result pages Poor No snapshot; inserts, deletes, or updates can shift pages and cause duplicates or omissions index.max_result_window defaults to 10,000
search_after Deep sequential pagination Good No snapshot by itself; a PIT provides a fixed index view for PIT-backed searches Requires deterministic sorting and a stable unique tie-breaker
scroll Bulk export and backfills Efficient for sequential extraction No snapshot isolation; later pages can be affected by index changes Search-context, heap, and file-handle overhead; not suitable for interactive navigation

OpenSearch Differences

  • OpenSearch supports from/size, search_after, and scroll with broadly comparable semantics.

  • OpenSearch 2.4 and later support point-in-time searches, allowing PIT-backed search_after pagination.

  • The default index.max_result_window is also 10,000 in OpenSearch. Confirm the effective value on the target index rather than relying on the default.

Production Guidance

  • Use from/size only for shallow navigation. Large offsets consume CPU and heap and eventually exceed index.max_result_window.

  • For deep user-facing pagination, use search_after with a PIT. Keep the PIT alive for the expected session and close it when finished.

  • Sort by fields that have doc values. Include a dedicated unique keyword or numeric field as the final tie-breaker; avoid using _id by default because sorting it can require expensive fielddata.

  • search_after is normally paged forward. Backward paging is also possible by reversing both the sort order and cursor direction, but it is not a random page-number API.

  • Use scroll for bounded exports or backfills, not interactive search pages. Set a keep-alive that covers the processing interval, consume batches promptly, and explicitly clear the scroll when finished.

  • Monitor search latency, rejected searches, heap pressure, and open search contexts. Inspect /_nodes/stats/indices/search and the task and scroll-management APIs during incidents.

  • Pagination work increases with the number of shards because the coordinator must merge per-shard results. For typical search workloads, avoid tiny shards and aim for primary shards around 10–50 GB, benchmarking against the actual query and reindex workload.

  • Use _cluster/health and _cat/shards when investigating degraded pagination. A yellow cluster has unassigned replicas and reduced redundancy but can normally serve searches; a red cluster has an unavailable primary and may return incomplete results.

Common Pitfalls

  • Requesting a page whose from + size exceeds index.max_result_window.

  • Omitting a unique, immutable tie-breaker, causing equal sort values to produce skipped or duplicate documents.

  • Sorting on _id or another high-cardinality field without doc values, increasing memory pressure.

  • Treating search_after alone as a consistent snapshot rather than combining it with a PIT.

  • Treating scroll as snapshot-isolated or leaving scroll contexts open after an export.

45

What is point in time (PIT) in Elasticsearch 8.x?

A point in time (PIT) is a server-side search context that provides a fixed view of searchable Lucene segments for a bounded lifetime. It enables consistent deep pagination with search_after while indexing, updates, and deletes continue.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x

How It Works

Creating a PIT opens a search context on the target shards and retains a reference to the Lucene segments visible at that time.

  • Fixed search view: Documents indexed, updated, or deleted after the PIT is created are not reflected in its results.

  • Refresh boundary: Only documents already visible through a completed index refresh are included. Creating a PIT does not make unrefreshed documents searchable.

  • Shard-level context: The PIT exists across the shards participating in the search. It is not an ACID-consistent transaction across multiple indices.

  • Normal search behavior: Mappings, analyzers, Query DSL filters, BM25 scoring, and aggregations continue to work normally.

  • Server-side state: The opaque PIT ID references retained server-side search contexts rather than copying the documents.

  • Expiry and renewal: keep_alive controls the context lifetime and can be extended by subsequent PIT searches.

  • Changing ID: Elasticsearch can return a different PIT ID in a later response. Always use the most recently returned ID for the next request.

PIT, Scroll, and from/size

Feature PIT Scroll from/size
Consistent result set Yes, within the PIT view Yes, within the scroll view No guarantee across separate requests
Deep pagination Uses search_after Uses the scroll cursor Large offsets consume resources and have practical limits
Search context Explicit and expiring Explicit and expiring No persistent context
Typical use Interactive deep pagination Controlled exports or scans Simple, shallow pagination
Operational cost Open contexts and retained segments Open contexts and retained segments Larger coordination and per-shard result overhead

PIT is generally preferable for interactive deep pagination. Scroll remains useful for processing a stable result set in a controlled export, although it should not be treated as a long-lived cursor.

OpenSearch Difference

OpenSearch supports PIT but exposes different REST resources. Elasticsearch creates a PIT with POST /<target>/_pit?keep_alive=... and closes it with DELETE /_pit. OpenSearch uses POST /<index>/_search/point_in_time?keep_alive=... and DELETE /_search/point_in_time.

Elasticsearch returns the created context as id, while OpenSearch returns pit_id; both are placed in the pit.id field during a search. Elasticsearch’s /_pit creation and deletion routes should not be copied directly into an OpenSearch deployment.

Production Best Practices

  1. Keep the lifetime short: Select a keep_alive longer than the expected interval between requests but short enough to release resources promptly. One to five minutes is a common interactive range.

  2. Track the latest ID: If a response returns a new pit_id, use it for the next request and for cleanup. Explicitly close abandoned PITs so contexts and retained segments are released promptly.

  3. Avoid excessive shard counts: A PIT adds context overhead for every target shard. A typical starting target is approximately 10–30 GB per shard, adjusted for ingest rate, retention, hardware, and query complexity. Thousands of tiny shards are especially expensive.

  4. Monitor production health: Track search latency, rejected tasks, JVM heap and GC pressure, disk usage, segment counts, and open search contexts. Use the PIT inspection APIs available in the installed 8.x release; GET /_cat/pits?v is available only in releases exposing the CAT PITs API.

  5. Plan for failures: PIT state is neither a backup nor a durable snapshot. If a node or affected shard fails, the context may become unusable. Recreate it and restart pagination, accepting that the new result set may differ.

  6. Protect search data: A PIT ID is not an authentication credential, but it references server-side resources and should not be exposed publicly. Continue using TLS, authentication, and index authorization.

  7. Use snapshots for recovery: Elasticsearch snapshots remain the appropriate mechanism for backups and disaster recovery; PIT provides search consistency, not data durability.

46

How do sorting and doc values work in Elasticsearch and OpenSearch?

Result sorting orders search hits by relevance, mapped fields, or special document identifiers. Doc values are Lucene’s disk-oriented, columnar field data that make sorting and many aggregations efficient without loading _source documents.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Query DSL Sorting

Elasticsearch and OpenSearch support several sorting forms:

  • Relevance sort: By default, results are ordered by _score descending.

  • Field sort: Sort by a keyword, numeric, date, IP, or other field that supports sorting.

  • Document order: _doc requests an inexpensive order based on Lucene document order within each shard.

  • Shard-local order: _shard_doc is a unique tie-breaker within a point-in-time context.

  • Missing values: missing: "_last" or missing: "_first" controls where documents without the field appear.

A relevance-first product search can sort by _score descending, then price ascending with missing prices last, and finally by a unique product_id for deterministic tie-breaking. track_scores is not needed when _score is explicitly a sort key because the score is already computed. It is useful when sorting primarily on other fields while requiring scores to be calculated and returned.

A keyword field is suitable for sorting identifiers or names. An analyzed text field is not suitable because it stores tokens rather than the original input. Add a keyword multi-field when both full-text search and exact sorting are required.

Avoid sorting directly by _id; copy its value into a keyword field, which uses doc values by default.

How Doc Values Work

  • Segment storage: Lucene stores doc values per field and per segment in compressed, column-oriented files.

  • Index-time preparation: Doc values are built as segments are indexed and flushed. They become available to searches when those segments are published by a refresh; segment merges consolidate them.

  • Efficient access: Numeric and date values use compact numeric encodings, while keyword values use term dictionaries and per-document ordinals.

  • Separate from _source: Doc values contain field values needed for sorting and aggregation, not the complete original JSON document.

  • Supported types: keyword, numeric, date, IP, and similar fields support doc values by default. text fields do not.

  • Fielddata fallback: Sorting or aggregating on an analyzed text field can use fielddata when explicitly enabled, but this loads values into the Java heap and can cause severe memory pressure. A keyword multi-field is preferred.

Doc values also power many aggregations, including terms, numeric, and date aggregations. The doc_values mapping option can disable them where storage savings matter, but doing so removes the efficient sorting and aggregation path.

Shards, Segments, and OpenSearch

Each primary shard is a Lucene index, and replica shards provide redundancy and additional search capacity. During a sorted search:

  1. The coordinating node sends the request to the relevant shards.

  2. Each shard searches its local segments and sorts its matching documents.

  3. The coordinator merges the shard results into the global ordering.

Doc values exist independently in every shard and replica. The core sorting and doc-value model is the same in Elasticsearch and OpenSearch.

Concern Elasticsearch 8.x OpenSearch 2.x/3.x
Sorting and doc values Lucene-backed shard-local values and coordinator merge Lucene-backed shard-local values and coordinator merge
Index lifecycle Index Lifecycle Management Index State Management
Vector search dense_vector fields and k-NN search dense_vector fields and the k-NN plugin
Distribution Elastic Stack tooling and licensing OpenSearch and OpenSearch Dashboards distribution

Related Search Behavior

  • Inverted index: Both products use Lucene’s inverted index to map analyzed terms to documents. Doc values supplement this structure; they do not replace it.

  • Mappings and queries: match searches analyzed text, term queries exact keyword values, and bool queries combine scoring must or should clauses with non-scoring filter clauses.

  • Relevance scoring: The default BM25 similarity considers term frequency, inverse document frequency, and field-length normalization. Sorting by _score orders by that relevance score.

  • Aggregations and ingestion: Doc values accelerate common aggregations, while both products provide ingest pipelines for parsing, enrichment, and indexing documents.

Production Practices

  • Shard sizing: Keep shards within a practical size, commonly tens of gigabytes rather than several hundred. Too many small shards increases cluster metadata and coordination overhead.

  • Cluster health: Green means all primary and replica shards are assigned; yellow means primaries are assigned but some replicas are not, which is normal on a single-node cluster configured with replicas; red means at least one primary is unavailable. Investigate allocation failures before sending production traffic.

  • Deep pagination: Avoid large from and size requests because the default index.max_result_window is 10,000. Use a point-in-time search with search_after and _shard_doc for stable deep pagination.

  • Monitoring: Track search latency, CPU, heap usage, garbage collection, disk latency, rejected tasks, unassigned shards, and cluster health. Sorting can increase CPU and I/O for large result sets.

  • Backups: Use the Elasticsearch or OpenSearch snapshot API with a supported repository, and test restores. Do not copy live data directories as a backup strategy.

  • Upgrades: Use rolling upgrades after checking compatibility. Mapping changes that cannot be applied in place require reindexing into a new index.

  • Security: Enable TLS and role-based access control, protect credentials outside Dev Tools requests, and restrict access to indices containing sensitive data.

47

What is fielddata in Elasticsearch, and why avoid it on text fields?

Fielddata is an in-memory data structure used by Elasticsearch and OpenSearch for operations such as aggregations and sorting. It is risky on text fields because it can consume substantial JVM heap memory and exposes analyzed tokens rather than the original field values.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Fielddata Works

Elasticsearch stores text using an inverted Lucene index. Each Lucene segment contains a term dictionary and postings that map terms to documents.

Some operations need field values rather than only postings. These include:

  • terms and significant_terms aggregations

  • Sorting on a field

  • Some scripts and bucket operations

For these operations, Elasticsearch may construct and load fielddata into the node’s JVM heap. Fielddata is organized by Lucene segment and managed through the node’s fielddata cache. It is transient rather than persisted, so it is rebuilt lazily on demand after a node restart.

Fielddata differs from _source:

  • _source stores the original JSON document.

  • doc_values stores field values in a compact, column-oriented format that aggregations can use efficiently.

  • fielddata stores values in heap memory in a form intended for term-oriented operations.

Why Text Fields Are Problematic

A text field is analyzed. For example, the value "Running shoes" might be indexed as the tokens running and shoes.

If fielddata is enabled on that field:

  • An aggregation returns individual analyzed tokens, not "Running shoes" as the original value.

  • The heap representation can be large, especially for frequently queried fields.

  • Large or numerous fielddata structures can trigger long garbage-collection pauses.

  • Elasticsearch may hit the fielddata circuit breaker and reject queries.

  • Heap pressure can cause node evictions, shard unavailability, or an unstable cluster.

  • Fielddata must be rebuilt after a node restart, potentially increasing work and memory pressure when it is requested again.

Text fields do not support doc_values. The preferred pattern is to use a keyword multi-field for exact-value aggregations while retaining the text field for full-text search.

Field type Typical use Aggregation storage Main concern
text Full-text matching Fielddata if explicitly enabled High heap usage, tokenized output, and Elasticsearch 8.x deprecation
keyword Exact values, filters, aggregations doc_values by default High-cardinality values can still consume resources

Recommended Design

Use a text field with a keyword subfield:

  • Analyze the text field for relevance scoring and full-text queries.

  • Aggregate or sort on the keyword subfield.

  • Set an appropriate ignore_above value if values longer than a configured character limit should be excluded from the keyword field.

  • Use doc_values for numeric, date, boolean, and keyword fields where supported.

For example, a product name can support both match queries and exact-term aggregations without enabling fielddata on the analyzed field.

Production Operations

Monitor more than whether an aggregation succeeds:

  • JVM heap usage and garbage-collection pause time

  • Fielddata memory usage and cache evictions

  • Circuit-breaker exceptions

  • Query latency and rejected requests

  • Shard allocation and cluster health

Use GET /_nodes/stats/indices to inspect node-level index and fielddata statistics, and use GET /_cluster/health to inspect cluster health. Avoid enabling fielddata during an incident merely to make a failing aggregation work; that can turn a query failure into a node-wide memory incident.

Elasticsearch and OpenSearch Differences

There is no material architectural difference in how fielddata works. Both systems use Lucene segments, disable fielddata on text fields by default, and provide doc_values on keyword fields.

Elasticsearch 8.x deprecates enabling fielddata on text fields; the preferred replacement is a keyword field or multi-field. OpenSearch 2.x still exposes the fielddata mapping option, but it has the same heap, circuit-breaker, and operational risks. The same design guidance applies to both systems.

48

How does the Elasticsearch Bulk API process requests?

The Elasticsearch Bulk API processes multiple index, create, update, or delete operations in one HTTP request using newline-delimited JSON (NDJSON). It reduces network overhead and improves ingestion throughput while allowing each operation to succeed or fail independently.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Request Format

Send the request to POST /_bulk or POST /<index>/_bulk with Content-Type: application/x-ndjson.

Each operation normally consists of two lines:

  1. An action line containing the operation and metadata.

  2. A source line containing the document, or no source line for delete.

Supported actions include:

  • index: Creates or replaces a document.

  • create: Creates a document only if its ID does not already exist.

  • update: Updates a document, optionally using doc_as_upsert.

  • delete: Removes a document.

When using /_bulk, specify the target index in each action’s metadata. When using /products/_bulk, the URL determines the index.

Processing and Responses

A coordinating node receives the request, parses the NDJSON, groups operations by target shard, and routes them to the appropriate primary shards. The primary coordinates the write and replication according to the index’s replica configuration.

Operations are ordered within a shard, but there is no global ordering guarantee across different shards. The client should therefore not depend on one shard’s operation completing before an operation targeting another shard.

The response is a JSON object containing one result per submitted operation under items:

  • errors: false means every item succeeded.

  • errors: true means at least one item failed, even when the overall HTTP status is 200.

  • Each item includes its own status and, on failure, an error object.

A request can fail as a whole. For example, malformed NDJSON commonly produces 400, an oversized payload produces 413, queue or thread-pool pressure can produce 429, and unavailable shards or overload can produce 503.

OpenSearch Differences

OpenSearch uses the same core _bulk NDJSON protocol and supports the same principal actions. The request syntax, item-level error handling, refresh behavior, and operational concepts are broadly compatible with Elasticsearch 8.x.

The main differences are product-specific rather than fundamental to the Bulk API: clients, plugins, security configuration, observability tools, and some advanced indexing features differ. Use the client and documentation matching the deployed OpenSearch version, and do not assume Elasticsearch-specific plugins or APIs are available in OpenSearch.

Code Example

The following creates two documents, updates one, and deletes another. It assumes the IDs do not already exist for the create operation.

POST /_bulk
Content-Type: application/x-ndjson

{"index":{"_index":"products","_id":"p-1001"}}
{"name":"Mechanical Keyboard","price":89.99,"tags":["input"]}
{"create":{"_index":"products","_id":"p-1002"}}
{"name":"USB-C Dock","price":129.50,"tags":["hardware"]}
{"update":{"_index":"products","_id":"p-1001"}}
{"doc":{"price":79.99},"doc_as_upsert":true}
{"delete":{"_index":"products","_id":"p-1002"}}

Do not add commas, blank separator lines, or a JSON array around the payload. The final source line must end with a newline.

For controlled visibility, refresh=wait_for waits for the next index refresh; refresh=true forces a refresh after processing. The default behavior does not make newly indexed documents immediately searchable.

Production Operations

  • Batch size: Start with moderate batches, commonly hundreds rather than very large payloads, and benchmark against the cluster. Excessively large requests increase latency, memory pressure, and the chance of rejection.

  • Shard sizing: Avoid both tiny and very large primary shards. A practical starting point is often roughly 20–50 GB per primary shard, but workload, document size, retention, and query patterns determine the correct value.

  • Backups: Bulk ingestion writes to primary shards and replicas, but it is not a backup mechanism. Use repository-backed snapshots for recovery and test restore procedures.

  • Monitoring: Track bulk latency, item failure rate, rejected requests, indexing rate, CPU, JVM garbage collection, disk watermarks, unassigned shards, and cluster health.

  • Failure handling: Retry failed items selectively rather than resubmitting the entire batch. Be careful with retries for non-idempotent operations, and use stable document IDs when possible.

  • Upgrades: Upgrade nodes according to the product’s rolling-upgrade requirements, monitor recovery and reindexing, and verify compatibility before changing mappings or analyzers.

  • Security: Use HTTPS, least-privilege write permissions, and avoid placing credentials in scripts or version control. Restrict bulk ingestion to trusted ingestion systems.

49

How to ingest data into Elasticsearch with Logstash, Beats, or pipelines?

Ingest pipelines run inside Elasticsearch nodes and are the lightest option, Beats are lightweight shippers on the edge, and Logstash is a separate heavy-duty processing service; choose by parsing complexity and throughput needs.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x (ingest pipeline syntax and Beats/Logstash output plugins differ slightly; noted inline).

How It Works

  • Ingest pipelines are defined in Elasticsearch and executed on ingest nodes during indexing. They use processors (grok, date, rename, set, script) to transform documents before they hit shards. Best for simple-to-moderate parsing with no extra infrastructure.

  • Beats (Filebeat, Metricbeat, Packetbeat, etc.) are single-purpose agents installed on source hosts. They tail files or collect metrics and send events directly to Elasticsearch or to Logstash. Minimal resource footprint.

  • Logstash is a JVM-based pipeline (input → filter → output) for heavy parsing, enrichment, and routing across many destinations. Use it when Beats cannot handle the transformation or when you need to fan out to multiple systems.

Comparison and OpenSearch Differences

Approach Runs Where Parsing Power Overhead Best For
Ingest pipeline Elasticsearch ingest node Medium Low (no extra infra) Simple transforms, default pipelines
Beats Source host Low Very low Edge collection, metrics, files
Logstash Separate service High High (JVM) Complex parsing, routing, enrichment
  • Ingest pipelines use the same processor names and REST paths in OpenSearch 2.x.

  • Beats can ship to OpenSearch only through the Elasticsearch output of Beats OSS 7.12.x or earlier (newer Beats refuse non-Elastic clusters); the opensearch output plugin is a Logstash plugin, not a Beats one.

  • Logstash uses the opensearch output plugin instead of elasticsearch when targeting OpenSearch.

Code Example

Create an ingest pipeline and attach it as the default for an index:

PUT _ingest/pipeline/apache-access
{
  "description": "Parse Apache combined access logs",
  "processors": [
    { "grok": { "field": "message", "patterns": ["%{COMBINEDAPACHELOG}"] } },
    { "date": { "field": "timestamp", "formats": ["dd/MMM/yyyy:HH:mm:ss Z"], "target_field": "@timestamp" } },
    { "remove": { "field": "message" } }
  ]
}
PUT logs-apache
{
  "settings": { "index.default_pipeline": "apache-access" }
}

Test before deploying with POST _ingest/pipeline/apache-access/_simulate and sample documents.

Filebeat ships log lines with minimal parsing; complex work is delegated to a pipeline or Logstash:

filebeat.inputs:
- type: log
  paths: ["/var/log/apache2/access.log"]
  fields: { service: apache }
output.elasticsearch:
  hosts: ["https://es.example.com:9200"]
  username: "filebeat_writer"
  password: "${FILEBEAT_PASSWORD}"
  ssl.certificate_authorities: ["/etc/pki/ca.crt"]

Use Logstash when you need multi-stage parsing, lookups, or routing to multiple outputs:

input { beats { port => 5044 } }
filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] }
}
output {
  elasticsearch {
    hosts => ["https://es.example.com:9200"]
    index => "logs-apache-%{+YYYY.MM.dd}"
    user => "logstash_writer"
    password => "${LOGSTASH_PASSWORD}"
  }
}

Best Practices and Pitfalls

  • Prefer ingest pipelines for new workloads; they scale with the cluster and avoid a separate JVM tier.

  • Keep Beats dumb: ship raw lines and parse centrally so parsing logic is version-controlled in one place.

  • Size ingest nodes: heavy grok or script processors need dedicated ingest nodes; monitor ingest.current and queue depth.

  • Handle backpressure: Beats and Logstash both buffer to disk; use persisted queues in Logstash and tune bulk sizes and queue limits.

  • Security: use dedicated least-privilege users (ingest_writer, beats_writer) with index-level privileges only; never ship credentials in plain config—use keystore or secrets.

  • Version skew: keep Beats and Logstash within one major version of the cluster to avoid mapping or API mismatches.

  • Test pipelines with _simulate and sample documents before rolling out to production indices.

50

What are ingest pipelines and processors in Elasticsearch?

An ingest pipeline is an ordered sequence of document-processing steps executed before indexing. Its processors transform, enrich, validate, or extract fields so the indexed document has the structure and types required by downstream mappings and queries.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Ingest Pipelines Work

Elasticsearch can select a pipeline in three ways:

  • The pipeline request parameter explicitly selects a pipeline.

  • index.default_pipeline applies a pipeline when the request does not select one.

  • index.final_pipeline applies a pipeline after the request or default pipeline and cannot be bypassed with _none.

An ingest node executes the selected processors in order. The transformed document is routed to the primary shard, which handles replication to its replica shards. Mappings and analyzers then determine how the document is stored and indexed.

Pipelines do not alter existing documents unless those documents are reindexed through the pipeline. The _simulate API can execute a pipeline against representative documents without writing them to an index, which is the preferred way to validate a definition before deployment.

Processors and Failure Handling

Common Elasticsearch processors include:

Processor Purpose
set Adds or replaces a field with a constant or copied value
date Parses a date and writes a normalized date value
grok Extracts fields using named regular-expression patterns
dissect Extracts fields using a fixed-delimiter pattern
rename Renames a field while preserving its value
remove Deletes a field
convert Converts a field between supported scalar types
script Runs a server-side Painless transformation
foreach Applies another processor to each element of an array
uri_parts Extracts URI components
geoip Adds geolocation data for an IP address
user_agent Parses a user-agent string
enrich Adds data retrieved from an Elasticsearch enrich index
inference Calls a configured inference service

Processor configurations commonly support if, ignore_failure, on_failure, description, and tag. If a processor fails without an error-handling option, indexing of that document is rejected. In a bulk request, other items can still succeed. ignore_failure continues with the next processor, while on_failure defines alternative processing for malformed documents.

Relationship to Indexing and Search

Elasticsearch distributes each index across primary shards and optional replica shards. Each shard contains Lucene indexes made up of immutable segments and an inverted index that maps analyzed terms to documents. An ingest pipeline runs before this process, allowing raw events to be normalized into fields that mappings can type correctly.

Analyzers affect indexed text and match queries; term queries use the exact indexed term. Within bool queries, filter clauses do not contribute to relevance scoring. BM25 supplies the default relevance scoring for applicable queries, while aggregations summarize matching documents.

Pipelines can standardize documents before data streams or index lifecycle management processes them, although ILM does not automatically run pipelines during lifecycle transitions. For vector search, pipelines may generate embeddings through the inference processor, while the destination mapping defines the appropriate dense_vector field. Heavy pipelines can therefore affect both ingestion capacity and query results.

OpenSearch Differences

OpenSearch supports the same core pipeline model and common processors such as set, date, grok, dissect, rename, remove, convert, script, and foreach. It also provides enrichment and plugin-specific ML or neural ingestion capabilities, but their availability and behavior depend on the OpenSearch release and installed plugins. Elasticsearch’s inference processor and index.final_pipeline must not be assumed to have equivalent OpenSearch implementations.

Production Practices and Common Pitfalls

  • Use the bulk API for sustained ingestion and avoid forcing a refresh for every event.

  • Monitor ingest rate and failures using GET /_nodes/stats/ingest, alongside JVM pressure, garbage collection, disk I/O, CPU saturation, and write-queue behavior.

  • Keep pipeline definitions in version control, review changes, and test them with _simulate against malformed and unusually large events.

  • Restrict pipeline-management privileges. Audit Painless processors because they can consume substantial CPU even though they run in Elasticsearch’s server-side scripting environment.

  • Store primary shards around 20–40 GB only as an initial search-workload guideline; benchmark actual indexing, query, recovery, and hardware requirements.

  • Remember that a green cluster has all primaries and replicas allocated, yellow means some replicas are unallocated, and red means at least one primary is unavailable.

  • Verify processor compatibility during rolling upgrades. Snapshots do not preserve pipeline definitions as reusable configuration, so redeploy them from source control after recovery.

51

How do you keep Elasticsearch synchronized with a primary database?

The standard production approach is change data capture (CDC): capture committed database changes and apply them to Elasticsearch asynchronously. This provides eventual consistency, so Elasticsearch remains a derived search index rather than the system of record.

Read full answer

Applies toElasticsearch 8.x, OpenSearch 2.x/3.x, and PostgreSQL 14–18

Sync Architecture

Elasticsearch consists of nodes and indices; each index has primary Lucene shards and optional replica shards. The database remains authoritative, while Elasticsearch serves search, filtering, aggregation, and vector queries.

Approach Characteristics Recommendation
Transactional outbox Commits the data change and event atomically Best default for transactional systems
WAL-based CDC Reads PostgreSQL logical decoding or similar database logs Lower database overhead and higher throughput
Application dual writes Writes to both systems independently Avoid; partial failures cause divergence
Scheduled full reload Rebuilds the index periodically Useful for reconciliation, not continuous synchronization

A robust flow is:

  1. Update the source row and increment its aggregate version in one transaction.

  2. Insert an immutable outbox event in that same transaction.

  3. Consume events in batches and index them through the Bulk API.

  4. Check every Bulk API item before marking only successful work complete.

  5. Reconcile Elasticsearch against the database periodically to detect missed or corrupt events.

Do not use the maximum outbox ID as a commit-order checkpoint: a lower ID can commit after a higher ID. Instead, use CDC commit offsets or claim rows with FOR UPDATE SKIP LOCKED, and retain enough state to replay failed work.

For migration, create a new versioned index, capture the change stream, backfill a consistent snapshot, replay concurrent changes, validate counts and samples, and then switch an alias.

Data and Search Model

  • Document identity: Use the database primary key as Elasticsearch _id.

  • Versioning: Store db_version and send it as the external version. external_gte permits an equal-version retry, so one immutable event per (aggregate_id, aggregate_version) is required.

  • Deletes: Send delete actions; a field update does not remove the document.

  • Index internals: An inverted index consists of immutable Lucene segments. Merges combine segments, while a refresh makes recent writes searchable.

  • Mappings and analysis: Define stable field types and analyzers. match performs analyzed full-text matching, term requires an exact value, bool combines clauses, and filter clauses do not contribute to scoring.

  • Relevance and aggregation: BM25 is the default relevance model. Bucket aggregations group results, while metric aggregations calculate values such as averages.

  • Ingestion and lifecycle: Ingest pipelines can normalize documents. Elasticsearch commonly uses Index Lifecycle Management (ILM); OpenSearch uses Index State Management (ISM). Retention and rollover policies must not delete data still needed for replay.

  • Vectors: Reindex when the embedding model changes because an updated document can otherwise retain a stale vector.

Code Example

This PostgreSQL outbox includes pending-work tracking:

CREATE TABLE search_outbox (
    id                bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    aggregate_id      text NOT NULL,
    aggregate_version bigint NOT NULL,
    event_type        text NOT NULL
        CHECK (event_type IN ('upsert', 'delete')),
    payload           jsonb,
    created_at        timestamptz NOT NULL DEFAULT now(),
    processed_at      timestamptz,
    UNIQUE (aggregate_id, aggregate_version),
    CHECK (
        (event_type = 'upsert' AND payload IS NOT NULL) OR
        (event_type = 'delete' AND payload IS NULL)
    )
);

CREATE INDEX search_outbox_pending_idx
    ON search_outbox (id)
    WHERE processed_at IS NULL;

The following REST request is valid in Kibana or OpenSearch Dashboards Dev Tools:

PUT /products-v1
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "id": { "type": "keyword" },
      "db_version": { "type": "long" },
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "price": {
        "type": "scaled_float",
        "scaling_factor": 100
      }
    }
  }
}

POST /products-v1/_bulk
{"index":{"_index":"products-v1","_id":"p-123","version":17,"version_type":"external_gte"}}
{"id":"p-123","db_version":17,"name":"Mechanical Keyboard","price":79.99}
{"delete":{"_index":"products-v1","_id":"p-124","version":18,"version_type":"external_gte"}}

Failure Modes and Operations

  • Bulk item failures: HTTP 200 can still contain failed items. Inspect each item’s status and error.

  • Version conflicts: With external_gte, a 409 means the incoming version is lower than the stored one, so treat it as a stale-event no-op. An equal version is accepted and overwrites the document, so never reuse a version number for a different payload.

  • Search visibility: Successful indexing is not immediately searchable until the next refresh.

  • Rate limiting: Retry 429 responses with bounded exponential backoff and jitter; route permanent 400 errors for correction.

  • Delivery semantics: Expect at-least-once delivery. Exact retries are idempotent only if the version and payload are immutable.

  • Ordering: External versions prevent older writes from winning. Route events by aggregate key when strict per-aggregate ordering is required.

  • Deletes and backfills: Emit delete events and combine a consistent snapshot with streamed changes.

Monitor outbox age, pending rows, consumer lag, failed bulk items, indexing latency, 429 and 409 rates, disk watermarks, heap pressure, shard count, and search errors. As a starting point, target roughly 10–50 GB per shard and validate through production testing; keep Elasticsearch heap below half of node RAM.

Best Practices

  • Keep the database authoritative and publish an explicit synchronization-lag metric.

  • Use versioned indices and aliases for mapping changes and rebuilds.

  • Tune Bulk API batches using production latency and failure data.

  • Use TLS, restricted network access, least-privilege credentials, and audited administration.

  • Retain unapplied outbox events and use snapshots when rapid restoration is valuable.

  • Test reconciliation, replay, failover, upgrades, and index rebuilds before relying on them during an incident.

52

What is index lifecycle management in Elasticsearch?

Index lifecycle management (ILM) automates how Elasticsearch indices age through storage tiers and are eventually deleted. It applies policies that roll over active indices, move older shards to less expensive tiers, adjust replicas, create searchable snapshots, and reclaim space according to retention requirements.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x (Index State Management)

How ILM Works

ILM manages an index through an ordered lifecycle policy. A policy is attached to an index, commonly through an index template, and its phases define what happens as the index ages.

  • Rollover: A rollover action creates a new write index when the current index meets a condition such as max_age, max_primary_shard_size, or document count.

  • Write alias: Applications normally write through an alias marked with is_write_index: true. Rollover updates that alias so the application does not need to know the physical index name.

  • Phase execution: After rollover, the old index progresses through later phases according to each phase’s min_age.

  • Shard allocation: Tier transitions can relocate shards between nodes, causing disk I/O, network traffic, and recovery work.

  • Deletion: The delete phase removes an index after its retention period. ILM itself is not a backup system.

Rollover creates another index; it does not split or rewrite the original index.

Lifecycle Phases and Actions

Phase Typical behavior Operational trade-off
hot Accepts writes and performs rollover Highest I/O and storage cost
warm Optimizes for occasional queries; read-only use can be configured Cheaper than hot tier, with possible slower searches
cold Uses lower-cost storage and generally avoids frequent queries Reduced query performance
frozen Uses low-cost nodes with data loaded on demand Lower local storage and memory footprint, but slower cache misses
delete Deletes the index Irreversible unless separate backups exist

searchable_snapshot is an action rather than a phase. It can be used in the cold or frozen phase to create a snapshot and search it through mounted snapshot data. Cold-tier snapshots may retain a local cache, while frozen-tier snapshots minimize local data copies. Data remains searchable on demand; no separate thaw operation is required.

forcemerge is optional. It can consume substantial I/O and CPU and is usually unnecessary for ordinary search workloads, although time-series data may benefit from controlled segment merging in a warm phase.

Policy Design and Trade-offs

A typical logs policy might roll over after one day or when the primary shard reaches approximately 50 GB, move the old index to warm tier after seven days, use cold or searchable-snapshot storage after 30 days, and delete it after 90 days.

  • max_primary_shard_size is generally more reliable than total index size because rollover is evaluated against the largest primary shard.

  • min_age is measured relative to rollover for an index managed by a rollover policy; for an index that has not rolled over, it is measured from creation.

  • The allocate action sets an absolute replica count. It reduces replicas only when the target is lower than the current replica count.

  • A searchable-snapshot action requires a registered snapshot repository and a successful snapshot operation.

  • Retention should be derived from operational, contractual, and regulatory requirements rather than copied from a tutorial.

A write alias and index template should be used together so both newly rolled-over indices and the initial index receive the required policy and rollover settings.

Monitoring and Failure Modes

Useful Elasticsearch APIs include GET _ilm/status, GET <index>/_ilm/explain, GET _cluster/health, and GET _cat/shards/<pattern>.

Important operational signals include:

  • Cluster health: green means all primaries and replicas are allocated; yellow means primaries are allocated but some replicas are not; red means at least one primary is unavailable.

  • Action failures: _ilm/explain reports the current step, failed_step, and failure information.

  • Allocation pressure: Disk watermarks, allocation filtering, or insufficient nodes can prevent tier transitions.

  • Shard movement: Large relocations can increase I/O, network usage, and recovery time.

  • Snapshot failures: Repository availability, permissions, expiration, and snapshot creation errors can block searchable-snapshot actions.

During incidents, operators should correct the underlying disk, allocation, snapshot, or write-blocking problem instead of repeatedly creating replacement policies.

Elasticsearch and OpenSearch Differences

Capability Elasticsearch 8.x OpenSearch 2.x
Feature Index Lifecycle Management (ILM) Index State Management (ISM)
Policy API _ilm/policy _plugins/_ism/policies
Lifecycle model Phases including hot, warm, cold, frozen, and delete Commonly hot, warm, cold, and delete
Snapshot support searchable_snapshot action in eligible phases snapshot action
Rollover model Managed through a rollover alias and policy Managed through an ISM rollover action

OpenSearch ISM supports comparable rollover, allocation, read-only, force-merge, snapshot, and deletion workflows, but its policy structure and state model differ. Elasticsearch ILM policies and APIs must not be copied directly into OpenSearch.

Security and Operational Practices

  • Back up separately: Configure a snapshot repository independently, monitor snapshot success, and test restores regularly before relying on retention-based deletion.

  • Apply least privilege: Restrict ILM or ISM, snapshot, monitoring, and index-management permissions with role-based access control.

  • Protect data in transit: Elasticsearch 8.x enables secured HTTP and transport communication by default; use TLS and do not expose the cluster directly to untrusted networks.

  • Size shards deliberately: Avoid both tiny shards, which increase cluster-state overhead, and oversized primary shards, which make recovery and relocation too slow. Validate sizes against measured throughput and acceptable recovery time.

  • Plan upgrades: Follow the supported rolling-upgrade sequence, retain enough temporary capacity for shard movement, and monitor recovery and disk watermarks.

  • Validate deletion: ILM and ISM deletion is not a backup mechanism; confirm that independent backups cover the full retention and recovery requirements.

53

What are Elasticsearch hot, warm, cold, and frozen data tiers?

Hot, warm, cold, and frozen data tiers are allocation levels that place index shards on nodes with different performance and cost characteristics. They are controlled through node roles and lifecycle policies, allowing frequently queried data to remain fast while rarely accessed data moves to cheaper storage.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x where noted.

How the Tiers Work

Each tier normally has a dedicated pool of nodes identified by roles such as data_hot, data_warm, data_cold, or data_frozen. Elasticsearch and OpenSearch allocate an index’s primary shards and replica shards to eligible nodes using allocation preferences.

Moving an index between tiers does not inherently compress it. Hot and warm tiers generally use local disks, while cold and frozen tiers use mounted searchable snapshots backed by remote object storage.

Tier Comparison

Tier Typical data Storage behavior Performance Typical use
Hot New or frequently updated data Local disk with high-performance storage Highest indexing and query performance Active logs, recent events, operational data
Warm Data still queried but no longer updated frequently Local disk on lower-cost nodes Lower write performance; reasonable query performance Historical logs, older transactional data
Cold Infrequently queried data Mounted searchable snapshot with limited local copies Higher latency, lower cost Compliance data, long-term analytics
Frozen Rarely accessed or archived data Searchable snapshot with cache-oriented, minimal local storage Highest latency and lowest infrastructure cost Archives, completed audits, very old telemetry

Cold and frozen tiers depend on a compatible searchable-snapshot repository, commonly S3-compatible object storage. Snapshot caches improve latency, but cache eviction can expose slow remote reads.

Allocation and Lifecycle

  • Elasticsearch: Index Lifecycle Management (ILM) uses the migrate action to move indices according to age, size, or other conditions. The searchable_snapshot action moves appropriate data into cold or frozen storage.

  • OpenSearch: Index State Management (ISM) and searchable snapshots provide similar functionality, although policies, action names, and implementation details differ.

  • Capacity requirement: If a node pool for the destination tier is unavailable, migration stops. Unassigned shards can turn cluster health yellow or red.

  • Shard behavior: Each tier stores the index’s Lucene shards and replicas according to the selected policy. Cold and frozen search may require loading data from shared storage when it is not cached.

Elasticsearch and OpenSearch Differences

Elasticsearch 8.x integrates tier allocation with ILM, searchable snapshots, and Stack Monitoring. OpenSearch exposes similarly named node roles and supports data-tier workflows through ISM and its searchable-snapshot implementation, but the products should not be treated as configuration-identical:

  • Elasticsearch ILM policies are not directly compatible with OpenSearch ISM policies.

  • Searchable-snapshot repository plugins and compatibility requirements differ.

  • Elasticsearch security features, encryption configuration, and monitoring tooling are not the same as OpenSearch Security and Performance Analyzer.

  • Benchmark both products with the actual repository, snapshot format, query mix, and expected cache-hit rate.

Production Best Practices

  • Size shards pragmatically: Hot shards around 10–50 GB are a common starting point, but workload and recovery-time objectives matter more than a fixed limit. Avoid thousands of tiny shards.

  • Preserve redundancy: Use at least one replica for important data when the tier has sufficient capacity. Remember that replicas consume both storage and I/O.

  • Plan headroom: Reserve CPU, disk, and I/O for rebalancing, merges, snapshots, replica recovery, and traffic spikes.

  • Monitor continuously: Track search and indexing latency, rejected requests, disk watermarks, unassigned shards, snapshot duration, object-store errors, and snapshot-cache hit rates.

  • Understand cluster health: Green means primaries and replicas are allocated; yellow generally means primaries are safe but replicas are unavailable; red means at least one primary is unavailable.

  • Test recovery: Regularly restore snapshots into isolated environments. A snapshot is not a complete backup strategy unless repository retention, security, immutability, and restoration procedures are also tested.

  • Secure every layer: Enable TLS and role-based access control, restrict snapshot repositories, encrypt remote data where required, and retain audit logs.

  • Upgrade incrementally: Follow a rolling-upgrade procedure and verify that node roles, searchable-snapshot repositories, and lifecycle policies remain compatible across versions.

54

What are data streams in Elasticsearch?

A data stream is an abstraction in Elasticsearch that manages a sequence of hidden backing indices as a single logical index, designed for append-only time-series data such as logs, metrics, and traces. Instead of writing to one index that grows unbounded, Elasticsearch automatically rolls over to new backing indices based on size, age, or document count, while searches transparently span all of them.

Read full answer

Applies toElasticsearch 8.x (concept also exists in OpenSearch 2.x with minor differences)

How Data Streams Work

  • A data stream is backed by multiple hidden indices named with a .ds-<data-stream-name>-<yyyy.MM.dd>-<generation> pattern.

  • Only the latest backing index (the write index) accepts new documents; older ones become read-only.

  • Searches against the data stream name automatically target all backing indices.

  • Rollover creates a new write index when conditions (max size, max age, max docs) are met, typically driven by an Index Lifecycle Management (ILM) policy.

  • Each document must contain a @timestamp field, which determines routing and time-based operations.

Requirements and Setup

To create a data stream you need an index template with data_stream: {} enabled and a mapping that includes @timestamp:

  • The template’s index_patterns must match the data stream name.

  • The mapping must define @timestamp as a date field.

  • ILM policy is attached via the template’s settings.

Data Streams vs Regular Indices

Aspect Data Stream Regular Index
Write target Latest backing index only The index itself
Rollover Automatic via ILM Manual or ILM on single index
Hidden backing indices Yes No
Requires @timestamp Yes No
Best for Append-only time-series Mutable documents, arbitrary updates
Document updates/deletes Not supported (append-only) Supported

Operational Considerations

  • Shard sizing: Each rollover creates a new set of shards. Plan primary shard count and size (target ~10-50 GB per shard) carefully; too many small shards waste resources.

  • Cluster health: Monitor green/yellow/red status per backing index; a single failed shard affects the whole data stream.

  • Retention: Use ILM delete phase to remove old backing indices; avoid frequent individual document deletes.

  • Updates/deletes: Data streams are append-optimized; in-place updates are not supported. For mutable data, use regular indices.

  • OpenSearch differences: OpenSearch 2.x supports data streams with similar semantics; ILM is replaced by Index State Management (ISM) policies, and some API paths differ.

Best Practices and Pitfalls

  • Always define @timestamp with a proper date format; missing it causes indexing failures.

  • Use ILM/ISM to automate rollover and deletion; manual rollover is error-prone.

  • Avoid high-cardinality fields in mappings to prevent mapping explosions.

  • Monitor backing index count and shard count; excessive rollover frequency creates too many shards.

  • In OpenSearch, use ISM policies instead of ILM and verify API compatibility.

55

How do you choose the number and size of shards in Elasticsearch?

Shard sizing should be driven by primary-data volume, expected growth, query fan-out, indexing throughput, and node-failure recovery time—not by a universal shard count. For ordinary production workloads, target approximately 10–50 GB per primary shard, often starting around 20–40 GB, and use as few primaries as the data distribution and parallelism requirements permit.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Principles

  • Primary shard: Contains one copy of the Lucene index. A shard count distributes indexing and query work but adds per-shard heap, metadata, file-handle, merge, and cluster-state overhead.

  • Replica shard: A complete copy of a primary on another node. Replicas provide availability and additional read capacity, but they do not store additional unique data and consume equivalent disk and resources.

  • Shard count: The primary count is fixed when an index is created; replica count can be changed later. An index can instead be replaced through split, shrink, reindex-and-alias, or rollover operations.

  • Shard size: Small shards waste resources and increase fan-out. Large shards lengthen recovery, relocation, and merge operations. A larger number of moderate shards generally recovers and scales more predictably than a few oversized shards.

Sizing Method

  1. Measure the representative primary Lucene footprint, not only the original JSON payload. Include inverted-index, doc-values, stored-field, segment, and delete overhead where measurable.

  2. Choose a target primary-shard size, normally around 30–40 GB for a starting point.

  3. Calculate primaries with ceil(primary_bytes / target_shard_bytes).

  4. Plan replicas separately. One replica is common in production, subject to disk and performance requirements.

  5. Validate the result against indexing rate, concurrent searches, merge load, heap, disk watermarks, and the time required to recover a failed node.

  6. Prefer several indices or a rolled-over data stream over creating one continuously growing index.

For example, 2 TiB of primary Lucene data divided by 30 GiB requires about 69 primaries. With one replica, the cluster holds about 138 shard copies and needs roughly 4 TiB of storage for those two copies, before additional segment overhead.

There is no universal maximum number of primaries per index or shards per node. Keep enough heap and disk for merges and relocation, avoid excessive fan-out, and benchmark the intended node-failure scenario. Replica-zero indices should normally be limited to disposable or temporary data whose authoritative copy exists elsewhere.

Resizing and OpenSearch

  • Split: Use the split API to increase the primary count, after satisfying its read-only or closed-index and routing prerequisites.

  • Shrink: Use the shrink API to reduce primaries. The source must be read-only or closed, and a single-shard target configuration may first be required.

  • Reindex and swap: Create a new index with the desired layout, reindex, validate it, then switch an alias.

  • Elasticsearch: ILM can roll a data stream when a primary reaches max_primary_shard_size.

  • OpenSearch: ISM policies provide rollover and deletion instead of Elasticsearch ILM. OpenSearch 2.x/3.x also provides split and shrink APIs, subject to the corresponding API prerequisites.

Code Example

For an index expected to contain about 150–200 GiB of primary data, five primaries produce shards of roughly 30–40 GiB. These creation and inspection requests work in Elasticsearch 8.x and OpenSearch 2.x/3.x:

PUT events-manual-000001
{
  "settings": {
    "index.number_of_shards": 5,
    "index.number_of_replicas": 1,
    "index.refresh_interval": "30s",
    "index.mapping.total_fields.limit": 2000
  },
  "mappings": {
    "properties": {
      "@timestamp": { "type": "date" },
      "host": { "type": "keyword" },
      "message": { "type": "text" },
      "service": { "type": "keyword" }
    }
  }
}

GET _cat/shards/events-manual-000001?v&h=index,shard,prirep,state,node,store,docs
GET _cluster/health?level=shards

Production Practices

  • Monitor shard sizes and allocation with GET _cat/shards, cluster health, node index statistics, search and write queue pressure, and merge activity.

  • Reserve disk capacity for merges and reallocation; keep usage below configured disk watermarks and allow at least one large shard to be relocated safely.

  • Use bulk requests of roughly 5–20 MB where throughput testing supports it, and avoid unnecessary updates and deletes.

  • Deleted documents are filtered from searches after the next refresh; Lucene merges reclaim their storage later. Soft deletes preserve history for active readers.

  • Take snapshots and rehearse restoration before resizing. A reindex-and-swap operation keeps the source until cutover and requires substantial temporary destination capacity.

  • During rolling upgrades, verify that the cluster can tolerate a node leaving while shards are simultaneously being initialized or relocated.

56

What problems do too many shards cause in Elasticsearch?

Too many shards in Elasticsearch (and OpenSearch) inflate cluster state and per-shard overhead, slow searches and recoveries, and can exhaust memory and file descriptors; the fix is usually fewer, larger shards sized around 10-50 GB each.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x.

How Shards Work

  • Each index is split into primary shards; each primary can have replicas.

  • Every shard is a separate Lucene index with its own segments, caches, and file handles.

  • The elected master publishes cluster-state updates, and every node receives and maintains the published state in memory; master-eligible nodes also participate in master-state consensus.

Problems Caused by Too Many Shards

  • Cluster-state bloat: More shards enlarge the state containing shard routing, mappings, and settings. Publishing and applying these updates adds work, especially on master-eligible nodes, and excessive state size can destabilize the cluster.

  • Fixed per-shard memory overhead: Even an empty shard consumes resources for buffers, caches, and segment metadata, so thousands of tiny shards waste RAM.

  • File-descriptor exhaustion: Each shard uses files for its Lucene segments and other data; hitting operating-system limits can cause indexing and search failures.

  • Search fan-out and latency: A query is coordinated across one copy of each relevant shard. More shards mean more scatter-gather work, higher CPU consumption, and greater tail latency.

  • Indexing and merge overhead: Each shard maintains its own indexing buffers and Lucene segments and performs refresh, flush, and merge work independently, increasing I/O and resource consumption.

  • Slow recovery and rebalancing: Node loss or scaling triggers relocation of many shards, extending recovery time and affecting availability.

  • Small-shard waste: Shards well under a few gigabytes carry proportionally high fixed overhead, resulting in poor storage and memory efficiency.

Practical Shard Sizing

Guideline Recommendation
Shard size Aim for approximately 10-50 GB per primary shard
Shards per node No more than roughly 20 shards per GB of heap
Maximum shards per node Default limit of 1,000 via cluster.max_shards_per_node
Total shard count Keep it as low as the data volume and recovery objectives allow

These are starting-point heuristics, not hard limits. Consider indexing throughput, query concurrency, retention, expected growth, and recovery-time objectives before finalizing shard counts.

Monitoring and Remediation

  • Monitor cluster health, pending tasks, heap usage, GC pressure, indexing latency, search latency, and file-descriptor usage. Deterioration may appear before the cluster reaches a hard failure.

  • Use rollover with ILM in Elasticsearch or ISM in OpenSearch to manage time-based indices and remove obsolete data rather than retaining many small indices indefinitely.

  • Consolidate indices with the shrink API or reindex them into fewer, larger shards. A shrink target must be a factor of the source index’s primary-shard count, and the source generally must meet the shrink API’s read-only requirements.

  • Use force merge sparingly, normally on read-only indices when reducing segment count is operationally necessary. It can cause substantial I/O and does not reduce the number of shards.

  • Delete or archive unused indices; an empty index can still consume cluster-state and shard-related resources.

OpenSearch Notes

  • OpenSearch has the same fundamental shard-sizing concerns. Use Index State Management (ISM) instead of Elasticsearch’s Index Lifecycle Management (ILM), and apply the same general sizing heuristics while accounting for OpenSearch-specific workload and recovery requirements.

57

How do you recover yellow or red Elasticsearch or OpenSearch clusters?

A yellow cluster has active primary shards but lacks assigned replicas, while a red cluster has at least one unavailable primary shard. Recovery starts by identifying the affected shard and the allocation decision preventing recovery, then correcting the underlying infrastructure or configuration problem.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Health Status Meanings

Status Meaning Operational impact
Green All primaries and replicas are assigned Normal operation
Yellow All primaries are active, but one or more replicas are unassigned Reduced redundancy; continued node failure can become an outage
Red At least one primary shard is unassigned and unavailable Partial indexing and search outage for affected indices

Elasticsearch and OpenSearch use Lucene shards. A replica is a complete copy of a primary shard on another node, so shard assignment, node topology and allocation settings directly affect cluster health.

Diagnosing the Allocation Failure

Start with the narrowest useful diagnostic:

  • Cluster health: Determine whether the problem affects the entire cluster or specific indices.

  • Shard state: Identify whether an unassigned shard is a primary or replica and whether it is initializing.

  • Allocation explain: Shows the deciders rejecting assignment, such as disk_threshold, awareness, same_shard, stale allocation filters or an unavailable node.

  • Node state: Check disk usage, heap pressure, long garbage-collection pauses, transport errors and nodes leaving unexpectedly.

  • Cluster settings: Look for disabled allocation, intentional node exclusions or rack/zone awareness that cannot currently be satisfied.

Do not restart every node indiscriminately. That can destroy the quorum needed to elect a master or cluster manager.

OpenSearch Differences

The health, allocation, shard and reroute APIs used here are compatible between Elasticsearch 8.x and OpenSearch 2.x/3.x. Operational differences primarily come from the distribution: OpenSearch deployments often use the OpenSearch Security Plugin, while managed services may restrict direct node, disk or cluster-setting changes. In OpenSearch 3.x, cluster manager terminology replaces older master terminology, although many compatibility APIs remain.

Code Example

Run these requests in Kibana Dev Tools, the OpenSearch Dashboards Dev Tools console, or another REST client:

GET /_cluster/health?level=indices&pretty

GET /_cat/nodes?v&h=name,node.role,heap.percent,ram.percent,disk.used_percent

GET /_cat/shards?v&h=index,shard,prirep,state,docs,store,node,unassigned.reason

GET /_cat/allocation?v&bytes=gb

GET /_nodes/stats/fs,jvm

POST /_cluster/allocation/explain
{}

GET /_cluster/settings?include_defaults=true

For a specific shard, replace the empty explain body with its actual index and shard:

POST /_cluster/allocation/explain
{
  "index": "orders",
  "shard": 0,
  "primary": true
}

After correcting the cause, ask Elasticsearch or OpenSearch to retry failed allocations:

POST /_cluster/reroute?retry_failed=true

If no usable copy remains and data loss is acceptable, allocate a stale primary explicitly:

POST /_cluster/reroute
{
  "commands": [
    {
      "allocate_stale_primary": {
        "index": "orders",
        "shard": 0,
        "node": "node-2",
        "accept_data_loss": true
      }
    }
  ]
}

Recovery Actions

  1. Resolve infrastructure failures

  2. Restore failed nodes and verify storage, filesystem permissions and network connectivity.

  3. Free or expand disk space when watermarks are blocking allocation.

  4. Check node-to-node TLS certificates, packet loss and high disk I/O latency.

  5. If the flood-stage watermark made indices read-only, allow disk usage to fall below the high watermark; the cluster normally removes the restriction automatically.

  6. Correct allocation constraints

  7. Review allocation awareness, node exclusions and cluster.routing.allocation.enable.

  8. If replicas cannot be placed because only one eligible node or rack exists, either add capacity or deliberately use zero replicas for an intentional single-node deployment.

  9. Retry allocation only after fixing the blocker.

  10. Recover a missing primary

  11. Prefer returning the original primary node so its replica can be resynchronized.

  12. If the original primary is permanently lost and an assigned replica exists, Elasticsearch or OpenSearch normally promotes that replica automatically.

  13. If no usable copy remains, use allocate_stale_primary or allocate_empty_primary with accept_data_loss: true only after an explicit recovery decision.

  14. Investigate corruption carefully

  15. Force merge does not repair missing or corrupted shards.

  16. Restore from a snapshot or reindex from the authoritative source when local recovery is impossible.

Production Practices

  • Alert on any red status, sustained yellow status, new unassigned shards, master or cluster-manager changes, disk watermarks and abnormal GC pauses.

  • Maintain tested snapshots, but do not treat snapshots as a substitute for replicas; restoration itself consumes cluster resources.

  • Use rolling upgrades and preserve quorum throughout the process.

  • Avoid hundreds of tiny shards; roughly 10–50 GB per shard is a common starting range for search workloads, then validate against ingestion rate, query pattern and recovery-time requirements.

  • Protect HTTP and transport endpoints with authentication, TLS and network controls. Never expose Elasticsearch or OpenSearch directly to the public internet.

58

What are unassigned shards in Elasticsearch and how do you diagnose them?

Unassigned shards are index shards for which the cluster currently has no assigned copy on any node. Missing replica shards produce yellow cluster health, while a missing primary produces red health and prevents operations requiring that shard.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

What Unassigned Shards Are

Every index is divided into primary shards, each with an optional set of replicas. If Elasticsearch cannot place a shard on an eligible node, that shard remains unassigned and the allocation service records the reason.

An unassigned primary makes its shard’s data unavailable to indexed operations, although other shards may remain available. An unassigned replica removes redundancy and reduces search capacity, but the primary can normally continue serving reads and writes until it fails.

Common Causes

  • Node loss or restart: shards formerly hosted by the departed node remain unassigned until they are recovered or reallocated.

  • Disk watermarks: the high watermark triggers shard relocation, while the flood-stage watermark blocks writes and prevents further allocation to the affected node.

  • Allocation filters: index settings such as index.routing.allocation.include, exclude, and require, or cluster allocation settings, can exclude eligible nodes.

  • Repeated allocation failures: persistent errors such as corrupt shard data can exhaust the configured allocation retries.

  • Replica placement constraints: a replica cannot be allocated to the same node as its primary, so replicas remain unassigned in a single-node cluster.

  • Cluster formation or master-election problems: a cluster without a valid elected master cannot reliably complete allocation.

Diagnosis Commands

Check health, including per-index and per-shard information:

GET _cluster/health?level=shards

List shard states and look for UNASSIGNED:

GET _cat/shards?v

Ask the cluster to explain an unassigned shard. With no body, Elasticsearch explains the first unassigned shard it finds:

GET _cluster/allocation/explain

Target a specific shard:

GET _cluster/allocation/explain
{
  "index": "logs-2024.06",
  "shard": 0,
  "primary": false
}

Inspect node disk allocation and filesystem statistics:

GET _cat/allocation?v
GET _nodes/stats/fs

Interpreting Allocation Explain

The response includes the shard’s current_state, unassigned_info, an overall allocate_explain, and—when node-level decisions are available—node_allocation_decisions. Each node entry has a node_decision (such as yes, no, or throttle) and a nested deciders array whose items carry decider, decision, and explanation.

Common deciders include:

  • disk_threshold: the node has insufficient disk space or violates a watermark.

  • same_shard: a replica would be placed on the primary’s node.

  • filter: an allocation filter excludes the node.

  • max_retry: the shard has exhausted its allocation retries.

  • awareness: the node’s awareness attributes do not permit placement.

Remediation and Best Practices

  • Fix the underlying cause before retrying allocation:

POST _cluster/reroute?retry_failed=true
  • Keep disk usage below the low watermark rather than approaching the default 95% flood-stage threshold. Monitor disk-used ratios and fs.total.available_in_bytes.

  • Treat 10–50 GB as a common starting range for shard size, then validate it against indexing rate, retention, query load, and recovery time. Avoid thousands of small shards.

  • Alert on non-green health and any sustained unassigned shards using the cluster health API or the exact metrics exported by the selected monitoring system.

  • During rolling upgrades and node restarts, allow a short grace period for recovery, but alert if unassigned shards persist.

  • Restrict _cluster/allocation/explain, _cluster/settings, and index-settings APIs to operators because they expose cluster topology and configuration.

  • Restore allocations promptly if they were temporarily disabled with cluster.routing.allocation.enable, and verify recovery afterward.

OpenSearch Notes

OpenSearch 2.x/3.x uses the same core shard-allocation model and the same _cluster/health, _cat/shards, and _cluster/allocation/explain APIs. The same workflow applies, although awareness rules, watermark configuration, and exported monitoring metrics should be checked against the specific OpenSearch deployment.

59

How do you scale Elasticsearch and OpenSearch clusters?

Elasticsearch and OpenSearch scale horizontally by distributing data and workloads across primary and replica shards on multiple cluster nodes. The right design balances data volume, indexing throughput, query concurrency, hardware capacity, and required fault tolerance.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Core Architecture

  • Cluster: A group of nodes sharing cluster state and coordinating shard allocation.

  • Index: A logical collection of documents divided into primary shards.

  • Shard: A Lucene index containing an inverted index and immutable Lucene segments. Merging segments consumes disk space and CPU.

  • Replica: A copy of a primary shard on another node. Replicas improve availability and read throughput but increase storage and indexing costs.

  • Failure domains: Place node copies across independent hosts, racks, or availability zones so one failure does not remove every copy.

The number_of_shards setting cannot be changed directly on an existing index. Use the split API to increase or the shrink API to decrease its shard count, treating both as expensive index rewrites. Update mappings and settings in place only when the changes are compatible; otherwise create a new index, reindex into it, and switch an alias.

Scaling by Workload

Requirement Primary scaling approach Trade-off
More data capacity Add data nodes and allow rebalancing Existing shard sizes do not automatically shrink
More read throughput Add replicas and optimize searches Replicas consume storage and must receive writes
More indexing throughput Add primaries through a new index and tune bulk ingestion More primaries increase cluster-state and merge overhead
Better query performance Filter early, use keyword fields, and bound aggregations Capacity is still needed for concurrent searches
Lower storage usage Apply retention, exclude unused fields from _source, or benchmark index.codec: best_compression Compression and retention affect CPU and recovery options

Choose primary-shard count from measured data size rather than document count. A common starting target is approximately 10–50 GB per shard, but indexing rate, query complexity, hardware, and recovery objectives determine the correct value. Avoid both tiny shards and a small number of oversized shards.

Indexing and Search Considerations

  • Mappings and analyzers affect index size and CPU. Use text for analyzed full-text search and keyword for exact filters, identifiers, and aggregations.

  • Query DSL separates scoring from filtering: match analyzes text, term performs exact lookup, and bool combines scoring must clauses with non-scoring filter clauses.

  • BM25 relevance uses term-frequency and document-level statistics. Aggressive analysis can improve matching while increasing index size and CPU usage.

  • Aggregations can consume substantial heap, especially with high-cardinality terms and broad ranges. Use filters, composite aggregations, and bounded date intervals.

  • Vector search adds index and memory pressure. Benchmark dimensions, similarity functions, filters, and replica requirements.

  • Ingestion should use appropriately sized bulk requests. Increase concurrency only while monitoring write queues, indexing latency, disk I/O, and rejected requests.

Code Example

This example creates an index with three primaries and one replica. Use it only when the measured shard count and capacity justify that layout.

PUT /products-v1
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "sku": { "type": "keyword" },
      "price": {
        "type": "scaled_float",
        "scaling_factor": 100
      }
    }
  }
}

A filtered full-text query can include a bounded aggregation:

GET /products-v1/_search
{
  "size": 10,
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "wireless headphones" } }
      ],
      "filter": [
        { "term": { "sku": "SKU-123" } }
      ]
    }
  },
  "aggs": {
    "price_ranges": {
      "range": {
        "field": "price",
        "ranges": [
          { "to": 50 },
          { "from": 50, "to": 100 },
          { "from": 100 }
        ]
      }
    }
  }
}

Production Operations

Monitor cluster health, node resources, shard balance, indexing and query latency, rejected requests, and circuit breakers:

GET /_cluster/health?pretty
GET /_cat/nodes?v
GET /_cat/shards?v
GET /_nodes/stats/breaker,indexing,os?pretty
POST /_cluster/allocation/explain
{}
  • Red health means at least one primary is unavailable; yellow usually means replicas are unassigned.

  • Crossing the flood-stage disk watermark can apply an index-level read-only block.

  • Heap pressure can trigger circuit breakers, rejected writes, and unstable searches.

  • Rebalancing and shard recovery can create substantial disk and network I/O.

  • Use tested snapshots for recovery rather than copying data directories directly.

  • Protect endpoints with TLS, private networking, RBAC, encryption at rest, and restricted snapshot-repository credentials.

  • Before upgrades, take a snapshot, review deprecations and compatibility, and follow the distribution’s supported rolling-upgrade procedure.

Best Practices and OpenSearch Differences

  • Benchmark with realistic mappings, document sizes, queries, aggregations, and indexing concurrency.

  • Use aliases and reindexing when changing incompatible shard layouts; do not expect automatic shard splitting.

  • Apply Elasticsearch ILM policies for rollover and retention. OpenSearch uses Index State Management (ISM) policies.

  • Elasticsearch uses Kibana, while OpenSearch uses OpenSearch Dashboards; dashboards, plugins, and APIs are not interchangeable.

  • Elasticsearch uses dense_vector and its k-NN APIs. OpenSearch commonly uses knn_vector with engine-specific implementations.

  • Both systems use primary/replica shards and Lucene, but lifecycle management, security, monitoring, and vector configuration require platform-specific validation.

60

How does Elasticsearch recover from node failures?

Elasticsearch recovers from node failure through leader election, replica promotion, shard reallocation, and peer recovery. In Elasticsearch 8.x the elected leader is the master; in OpenSearch it is the elected cluster manager.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x and 3.x

Recovery Sequence

  1. A data node fails

  2. The shards allocated to that node become temporarily unassigned.

  3. The elected leader updates cluster state and coordinates shard allocation.

  4. If an in-sync replica exists on another node, it is promoted to primary. Writes previously replicated to that replica remain available, subject to the index’s replication and durability guarantees.

  5. If the failed node returns, its stale shard copy is normally recovered and attached as a replica rather than automatically replacing the promoted primary.

  6. A leader-eligible node fails

  7. The remaining voting nodes detect the failure and hold an election.

  8. A candidate must obtain the required voting quorum before becoming the elected master or cluster manager.

  9. Three voting nodes normally require two votes, and five require three. This prevents two disconnected groups from independently publishing cluster state and accepting conflicting writes.

  10. Without quorum, cluster-state updates and writes stop. Some nodes may still serve potentially stale reads from local data.

  11. Redundancy is restored

  12. The allocator searches for another eligible node with sufficient disk space and compatible allocation settings.

  13. The replacement shard copy is initialized through peer recovery, which transfers Lucene segment files from the source copy.

  14. The target replays the relevant translog operations and becomes an in-sync replica.

  15. Recovery consumes disk bandwidth and network capacity, so multiple simultaneous failures can reduce indexing throughput and increase latency.

Failure Modes

  • No surviving replica: The primary shard remains unassigned and the index becomes red until the original node returns, a snapshot is restored, or another copy becomes available.

  • Loss of leader quorum: The cluster cannot elect a leader or publish normal cluster-state changes, so writes and most administration operations stop.

  • Disk pressure: High disk watermarks prevent shard allocation. At flood-stage watermarks, Elasticsearch and OpenSearch can mark affected indices read-only to protect the cluster from disk exhaustion.

  • Network partition: A partition that cannot retain voting quorum is deliberately prevented from accepting writes, avoiding split brain.

  • Complete loss of every copy: Replicas do not protect against simultaneous loss of all shard copies. The translog is a recovery mechanism, not a backup.

  • Accidental deletion or malformed data: Replicas replicate destructive changes as well. External version IDs can reduce accidental overwrites, but point-in-time recovery requires a suitable snapshot.

Elasticsearch and OpenSearch Differences

Area Elasticsearch 8.x OpenSearch 2.x/3.x
Leader terminology Master-eligible and elected master Cluster-manager-eligible and elected cluster manager
Main configuration file elasticsearch.yml opensearch.yml
Security Built-in TLS, role-based access control, and audit capabilities Security is provided by the OpenSearch Security plugin
Recovery model Leader election, primary promotion, allocation, and peer recovery The same core model

Snapshot repositories, plugins, APIs, and operational defaults differ, but both products recover failed-node shards by promoting replicas and allocating replacement copies.

Monitoring and Production Practices

  • Cluster health: green means all shards are assigned, yellow means all primaries are assigned but some replicas are not, and red means at least one primary is unassigned.

  • Diagnosis: Use _cat/nodes to inspect node state, _cat/shards to find unassigned copies, and the cluster allocation explain API to determine why allocation is blocked.

  • Capacity monitoring: Track disk usage and latency, JVM heap pressure, request rejections, translog growth, recovery throughput, and unassigned-shard counts.

  • Leader configuration: Prefer three leader-eligible nodes across independent failure domains. Five may be justified for larger installations, but every additional voting node increases quorum requirements.

  • Replica strategy: Use at least one replica for availability during single-node maintenance. Configure zone or rack awareness so primary and replica copies are not concentrated in one failure domain.

  • Shard sizing: Size shards from measured data, ingest rate, and recovery-time objectives. Avoid both very large shards, which recover slowly, and excessive shard counts, which burden the cluster manager and heap.

  • Maintenance: Perform rolling upgrades and restarts one node at a time while preserving replica capacity and voting quorum.

  • Backups: Store snapshots outside the cluster, monitor snapshot failures separately, and regularly test restoration. Index lifecycle management can automate retention but does not replace backups.

  • Security: Use TLS, least-privilege roles, restricted cluster-management access, and protected snapshot credentials.

61

How do I back up and restore Elasticsearch using snapshots?

Snapshots are incremental, shard-level backups managed by Elasticsearch or OpenSearch. You register a repository, create snapshots in it, and restore selected indices from a successful snapshot without copying Elasticsearch’s data directory.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Snapshots Work

  • Repository: A named location such as shared filesystem, Amazon S3, Google Cloud Storage, or Azure Blob Storage. Repository type and location must be consistent when restoring.

  • Incremental storage: The repository reuses unchanged data files from earlier snapshots and stores only newly required data, reducing backup time and storage.

  • Shard-level operation: Each shard is processed independently, so duration and throughput depend on shard count, shard size, repository performance, and source-node I/O.

  • Captured state: Index data, mappings, index settings, and aliases are stored. Cluster-wide metadata is included only when include_global_state is true; restoring it can overwrite current settings, templates, policies, and security-related metadata.

  • Consistency: A snapshot provides a point-in-time view of each index, but it is not transactionally consistent across unrelated indices.

Keep the repository in a separate failure domain. A repository on the same hosts as the cluster is not a complete disaster-recovery backup.

Operational Behavior

  • Snapshot states include IN_PROGRESS, SUCCESS, PARTIAL, and FAILED. A PARTIAL snapshot has missing or failed shard data and should not be treated as a complete recovery point.

  • Restored indices inherit their original replica count. On a single-node recovery cluster, override index.number_of_replicas to 0; otherwise the cluster remains yellow.

  • To overwrite an index, close the existing target before restoring. Check the restored index state and recovery status, and open it only if it is still closed. Existing aliases can also cause restore conflicts.

  • For search-heavy workloads, 10–50 GB per shard is a practical starting range. Avoid very large shards and thousands of tiny shards because recovery and cluster-state operations become inefficient.

  • Monitor snapshot state and duration, shard failures, repository latency and errors, recovery progress, disk pressure, and source-node CPU and I/O.

Elasticsearch and OpenSearch

Area Elasticsearch 8.x OpenSearch 2.x/3.x
Scheduling Snapshot Lifecycle Management (SLM) ISM policy with the snapshot action
Common repositories fs, s3, gcs, azure fs, s3, gcs, azure
Global state Controlled by include_global_state Controlled by include_global_state
Compatibility Restore only into a version supported by the Elasticsearch compatibility matrix Restore only into a version supported by the OpenSearch compatibility matrix

The basic repository, snapshot, status, and restore REST calls below overlap across the products. Their snapshot repository formats are product-specific, however, so do not attempt to restore an Elasticsearch repository into OpenSearch or vice versa.

Code Example

For each product, mount a separate repository directory at /mnt/backups/search on every node. The path must resolve to the same shared filesystem from every node. Add the setting to each product’s configuration file and restart the service:

# elasticsearch.yml or opensearch.yml
path.repo: ["/mnt/backups/search"]

Register and verify the repository:

PUT /_snapshot/backup_repo
{
  "type": "fs",
  "settings": {
    "location": "/mnt/backups/search",
    "compress": true
  }
}

POST /_snapshot/backup_repo/_verify

# Start an asynchronous snapshot.
PUT /_snapshot/backup_repo/orders-2025-03-08?wait_for_completion=false
{
  "indices": "orders-*",
  "include_global_state": false
}

GET /_snapshot/backup_repo/orders-2025-03-08/_status
GET /_snapshot/backup_repo/orders-2025-03-08

# Restore one index into a cluster where it does not already exist.
POST /_snapshot/backup_repo/orders-2025-03-08/_restore
{
  "indices": "orders-2025-03-01",
  "include_global_state": false,
  "include_aliases": true,
  "index_settings": {
    "index.number_of_replicas": 0
  }
}

GET /_cluster/health/orders-2025-03-01?wait_for_status=green&timeout=2m
GET /orders-2025-03-01/_count

The replica override is appropriate for the single-node example; set the intended replica count explicitly when restoring into a larger production cluster.

Production Practices

  • Test recovery: Restore periodically into an isolated cluster and validate counts, mappings, aliases, representative searches, and application behavior.

  • Protect data: Encrypt data in transit and at rest, restrict snapshot and repository-management permissions, and place object-store credentials in the product’s secure-settings mechanism.

  • Apply retention: Use SLM or ISM policies to remove obsolete snapshots while preserving required recovery points.

  • Control impact: Schedule snapshots away from peak ingestion and watch repository throttling, disk pressure, and source-node saturation.

  • Manage global state: Decide deliberately whether each snapshot includes global state, and back up excluded cluster configuration and security artifacts separately.

  • Plan upgrades: Verify product, mapping, and plugin compatibility before upgrades. Keep the previous version available until a tested restore succeeds.

  • Separate failures: Copy finalized snapshots to another secured repository or bucket with independent retention controls and a documented repository-recovery procedure.

62

How do you secure Elasticsearch and OpenSearch in production?

Defense in depth is the standard approach: combine network isolation, TLS, strong authentication, least-privilege authorization, encryption at rest, audit logging, and operational monitoring. Neither Elasticsearch nor OpenSearch should rely on a firewall or hidden cluster address as its only protection.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x with the Security plugin

Security Architecture

Security boundaries exist at several levels:

  • Cluster: Cluster permissions control operations such as monitoring, index management, snapshots, and security administration.

  • Node: Elasticsearch and OpenSearch run different node roles, such as ingest, coordinating, data, and master nodes. Restrict administrative traffic to master nodes.

  • Index: Index privileges determine which applications can read, write, create, or administer specific indices or patterns.

  • Document and field: Elasticsearch provides document- and field-level security in licensed tiers (Platinum/Enterprise); OpenSearch provides it in the Security plugin.

  • Shards and replicas: Shards are storage units, not authorization boundaries. Replicas contain the same data and must receive identical security and encryption protections.

Use separate credentials, service accounts, or API keys for Kibana, Beats, CI/CD, backups, and application traffic. Avoid sharing the superuser or admin role between systems.

TLS and Network Controls

Enable TLS on both network paths:

  • Transport traffic between Elasticsearch or OpenSearch nodes protects replicated data and cluster coordination.

  • HTTP traffic protects Kibana, ingest clients, monitoring tools, and API consumers.

For Elasticsearch 8.x, configure xpack.security.transport.ssl.enabled: true and xpack.security.http.ssl.enabled: true, using configured keystores and truststores. For OpenSearch 2.x, configure plugins.security.ssl.transport.enabled: true and plugins.security.ssl.http.enabled: true, with the appropriate PEM certificate, private-key, and trusted-CA paths.

Use separate certificates and private keys for HTTP and transport. Protect private keys with the platform secret manager or the product’s secure keystore; do not commit them to Git or store them in ordinary configuration files.

Never expose ports 9200 or 9300 publicly. Restrict them to private subnets such as 10.0.0.0/16, enforce host firewalls or security groups, and place client access behind an authenticated proxy or load balancer. Configure CORS with explicit allowed origins rather than wildcards.

Authentication and Authorization

Use a layered authentication strategy:

  • Elasticsearch: Native users for break-glass administration, API keys for automated clients, service accounts for Elastic Stack components, and an external identity provider through supported OIDC or SAML integrations.

  • OpenSearch: Security-plugin internal users, backend roles, supported OIDC or SAML integrations, and managed-service IAM where available.

Apply least privilege:

  • Grant only required cluster and index privileges.

  • Restrict index patterns to the narrowest usable scope.

  • Separate read-only dashboards from ingestion and administration identities.

  • Give CI/CD systems permission to deploy mappings without unrestricted cluster administration.

  • Review direct grants, API keys, role mappings, and users regularly.

A 401 response normally indicates missing or invalid authentication, while 403 indicates that an authenticated principal lacks sufficient authorization.

Data Protection and Auditing

  • Encrypt node disks and replicas using KMS-backed volumes or equivalent platform encryption.

  • Encrypt snapshots separately, including repository credentials and storage-provider encryption settings.

  • Do not assume that an Elasticsearch snapshot or an OpenSearch index is encrypted merely because TLS is enabled.

  • Store snapshots and security configuration in separate protected locations.

  • Test restoration regularly; a snapshot that has never been restored is not a proven backup.

  • Enable security audit events and forward them to a central SIEM. In Elasticsearch, set xpack.security.audit.enabled: true. In OpenSearch, set plugins.security.audit.type (for example internal_opensearch) and configure the audit settings accordingly.

  • Monitor privilege changes, role mappings, API-key creation, authentication failures, and repeated 401 or 403 responses.

Audit logs can contain request metadata and sometimes sensitive payloads, so restrict their access and retention appropriately.

Code Example

The following examples create narrowly scoped access. Run them as an administrator through Kibana Dev Tools.

Elasticsearch 8.x:

PUT /_security/role/logs-reader
{
  "cluster": ["monitor"],
  "indices": [
    {
      "names": ["logs-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ]
}

POST /_security/api_key
{
  "name": "logs-reader",
  "role_descriptors": {
    "reader": {
      "cluster": ["monitor"],
      "indices": [
        {
          "names": ["logs-*"],
          "privileges": ["read", "view_index_metadata"]
        }
      ]
    }
  }
}

The API-key response contains an encoded credential that is shown only once; store it in a secret manager and rotate it regularly.

OpenSearch 2.x:

PUT /_plugins/_security/api/roles/logs-reader
{
  "cluster_permissions": ["cluster_monitor"],
  "index_permissions": [
    {
      "index_patterns": ["logs-*"],
      "allowed_actions": ["read"]
    }
  ]
}

PUT /_plugins/_security/api/internalusers/sre
{
  "password": "REPLACE_WITH_SECRET",
  "backend_roles": ["sre_team"]
}

PUT /_plugins/_security/api/rolesmapping/logs-reader
{
  "backend_roles": ["sre_team"],
  "users": ["sre"]
}

Production Operations and Failure Modes

  • Monitor certificate expiry, clock synchronization, authentication failures, disk watermarks, JVM memory, shard allocation, snapshot failures, and security-index availability.

  • Use GET /_cluster/health after security or topology changes. yellow may be expected on a single-node cluster with replicas, but red requires immediate investigation.

  • Keep shards generally in the practical 10–50 GB range and avoid creating very large numbers of tiny shards. Replicas improve availability but consume additional storage and indexing resources.

  • During rolling upgrades, distribute new CA certificates before rotating node certificates. Verify role compatibility and avoid disabling authentication to bypass an incident.

  • Back up the Security index with cluster snapshots, while separately protecting certificates, role definitions, and configuration required to restore access.

  • Treat expired certificates, mismatched trust chains, full disks, lost encryption keys, incorrect role mappings, and leaked API keys as separate incident scenarios with dedicated runbooks.

63

What JVM heap size should Elasticsearch use, and why 31 GB?

Elasticsearch should use a JVM heap of at most about 30–31 GB, sized to roughly half of the node’s RAM, because crossing ~32 GB disables compressed ordinary object pointers (compressed oops) and roughly doubles the per-reference memory cost.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x (JVM heap tuning)

Why ~31 GB Is the Ceiling

  • The JVM can address heap with 4-byte compressed oops instead of 8-byte pointers when the heap stays below ~32 GB.

  • Below that threshold, object references cost half as much, so a 30 GB heap holds far more live data than a 32 GB heap with 8-byte pointers.

  • Setting heap above ~31 GB therefore wastes RAM: you pay for 8-byte references without gaining proportional capacity.

  • Leave headroom below 32 GB (commonly 30g or 31g) so JVM internals and compressed-oops alignment never push you over the threshold.

Recommended Sizing

  • Set heap to no more than 50% of physical RAM; the rest is for the OS filesystem cache, which Lucene relies on heavily.

  • Always set -Xms equal to -Xmx to avoid heap resizing and GC churn at runtime.

  • Prefer adding nodes over raising heap beyond ~31 GB.

  • Elasticsearch 8.x defaults to the G1 garbage collector; keep GC logs enabled for diagnosis.

Production Guidance

  • Monitor heap usage, GC pause times, and circuit-breaker trips; sustained heap above ~75% is a warning sign.

  • Use indices.breaker.total.limit and request circuit breakers to protect nodes from OOM during heavy aggregations.

  • For upgrades, change heap in jvm.options or via ES_JAVA_OPTS, restart the node, and verify cluster health returns to green before proceeding.

Common Pitfalls

  • Setting heap to 50%+ of RAM and starving the OS page cache, hurting search latency.

  • Using different -Xms and -Xmx values, causing runtime resizing pauses.

  • Raising heap past ~31 GB expecting linear scaling, then seeing worse GC behavior and wasted memory.

64

How do you speed up Elasticsearch and OpenSearch indexing?

Bulk ingestion, controlled refresh frequency, right-sized shards, and reduced write amplification provide the largest indexing gains. These improvements require benchmarking against representative data while preserving durability, replication, and cluster stability.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Indexing Works

Each document is routed to a primary shard and copied to replica shards. The primary runs the mapping analyzers, tokenizes text, and writes terms to a Lucene inverted index.

Two separate mechanisms affect indexing:

  • Refresh: Makes recent writes visible to search by opening a new searcher. It controls search visibility, not durability, and does not necessarily create a new Lucene segment.

  • Translog: Records operations so they can be recovered after a node failure. It affects durability and disk-write latency.

Frequent refreshes add refresh and searcher overhead. When indexing buffers are flushed into many small segments, later merges consume CPU and disk I/O. Replicas, updates, document size, mapping complexity, and insufficient disk or CPU can also become bottlenecks.

Highest-Impact Techniques

Technique Practical guidance Trade-off
Use the Bulk API Send roughly 1,000–5,000 documents or 5–15 MB per request. Excessively large requests increase latency and memory use.
Control refresh frequency Temporarily use 30s or -1 during a large initial load, then restore the normal interval. Documents remain invisible to search until a refresh occurs.
Right-size shards Start around 10–30 GB per shard for write-heavy data and benchmark. Too many shards create overhead; too few create oversized merge and recovery work.
Limit write amplification Temporarily remove replicas only when redundancy is not required. A node failure can lose unavailable data when replicas are absent.
Avoid unnecessary indexing Set index: false for fields that only need retrieval from _source. Those fields cannot be searched, aggregated, or sorted normally.
Avoid frequent updates Use append-only events and deterministic document IDs. Updating a document leaves the old version to be cleaned up during merges.
Bound client concurrency Increase concurrency gradually and back off on 429 or temporary 503 responses. Unbounded parallelism can overwhelm the write thread pools and disk.

Prefer a stable, explicit mapping over dynamic mapping. Use keyword for identifiers and exact filters, and text only when full-text analysis is required. Avoid field explosion, very large documents, and disabling _source globally. The latter saves storage and transfer costs but breaks features such as reindexing from the existing source and makes many recovery workflows harder.

Code Example

The following example creates a temporary write-optimized index. Three primary shards are only an example; determine the count from data volume, node count, and benchmark results.

PUT /logs-fast
{
  "settings": {
    "index": {
      "number_of_shards": 3,
      "number_of_replicas": 0,
      "refresh_interval": "30s"
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "timestamp": { "type": "date" },
      "service": { "type": "keyword" },
      "message": { "type": "text" }
    }
  }
}

Submit data in newline-delimited JSON. refresh=false avoids forcing a refresh after every request; it does not disable periodic refreshes.

POST /logs-fast/_bulk?refresh=false
{"index":{"_id":"evt-001"}}
{"timestamp":"2025-01-15T10:00:00Z","service":"checkout","message":"Order accepted"}
{"index":{"_id":"evt-002"}}
{"timestamp":"2025-01-15T10:00:01Z","service":"checkout","message":"Payment completed"}

Make documents searchable explicitly after the load:

POST /logs-fast/_refresh

Always inspect the Bulk API response for errors: true and per-item failures. A successful HTTP response can still contain rejected individual operations.

Monitoring and Failure Modes

Useful monitoring requests include:

GET /_cat/indices/logs-fast?v=true
GET /_nodes/stats/indices,indexing,fs
GET /_cat/thread_pool/write?v=true

Track ingestion latency, write queue depth, rejected operations, CPU, disk latency, segment count, merge pressure, translog size, heap usage, and cluster health.

  • Red cluster: At least one primary shard is unavailable; investigate immediately.

  • Yellow cluster: Replicas are unassigned, which is common in a single-node test but risky in production.

  • 429 responses: Apply exponential backoff with jitter rather than immediately retrying.

  • 409 conflicts: Concurrent updates targeted the same document version.

  • Disk watermark violation: Elasticsearch may mark shards read-only to prevent corruption.

  • Node restart: After an unclean stop, recovery replays the translog; large uncheckpointed translogs and many replayed operations increase recovery time. Refresh frequency does not truncate the translog or control recovery time.

Use snapshots for backup, but remember that snapshots are incremental and do not replace replica-based high availability.

Security, Lifecycle, and Vectors

Protect ingestion endpoints with TLS, authentication, and least-privilege roles. Load secrets from a secret manager or environment variables rather than embedding credentials in scripts or source control. Also enforce request-size limits at trusted proxies.

Use ILM or OpenSearch Index State Management to roll over oversized indices and delete data according to retention policy. Move cold data only through the storage mechanism your deployment supports, such as Elasticsearch data tiers or searchable snapshots, or OpenSearch cold-tier nodes and snapshots. Coordinate lifecycle transitions with ingestion benchmarks because merges, deletes, snapshots, and segment optimization all consume resources.

Vector ingestion has additional costs: HNSW graph construction increases memory use and indexing work. Benchmark vector dimensions, graph parameters, memory limits, and recall before enabling a large vector workload.

Elasticsearch vs OpenSearch

The Bulk API, mappings, refresh settings, shard strategy, and monitoring principles are broadly similar in Elasticsearch 8.x and OpenSearch 2.x. Vector mappings and tuning controls differ significantly: Elasticsearch uses dense_vector and HNSW options, while OpenSearch configurations depend on its k-NN plugin and selected engine. Do not copy vector mappings between products without validating them. In both systems, measure results with production-like documents, mappings, concurrency, retention, and failure conditions before changing production settings.

65

How can I optimize search queries in Elasticsearch and OpenSearch?

Query optimization in Elasticsearch and OpenSearch combines efficient mappings, selective queries, appropriate shard counts, and continuous profiling. The largest gains usually come from eliminating broad wildcard searches, moving non-scoring conditions into filters, and controlling expensive aggregations.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

How Search Executes

Elasticsearch and OpenSearch distribute searches across a cluster of nodes. A coordinating node selects an eligible primary or replica copy of each relevant shard, merges their results, and returns them to the client.

  • Inverted index: Maps analyzed terms to document lists, locating matches without scanning every document.

  • Lucene segments: Immutable segments contain indexed data. Normal merging controls segment growth and removes deleted-document overhead.

  • BM25: The default relevance model in both engines considers term frequency, inverse document frequency, and field length.

  • Filter context: Clauses in a bool.filter do not contribute to the score and are eligible for query-cache reuse.

  • Replicas: Improve availability and read capacity, but querying a replica does not inherently reduce the work required to produce and merge results.

Optimize Mappings and Queries

  • Use text with a keyword subfield when a field needs full-text and exact matching. Use keyword directly for identifiers, statuses, and categories.

  • Apply compatible analyzers to indexed text and search values. A term query normally should not target an analyzed text field.

  • Prefer match for analyzed text and term for exact keyword, numeric, boolean, or date values.

  • Put non-scoring conditions in bool.filter. Put only relevance-producing conditions in must or scoring should clauses.

  • Avoid leading wildcards, broad regular expressions, and user-supplied query_string expressions; they can enumerate or examine many terms.

  • Restrict _source, set a bounded size, and use "track_total_hits": false when an approximate total is acceptable.

  • Use composite aggregation pagination for large result sets. High-cardinality terms and cardinality aggregations can consume substantial heap.

Tune Index and Cluster Design

  • Shard sizing: Roughly 10–50 GB per shard is a common search-workload starting point, but query shape, cardinality, hardware, and recovery objectives matter more.

  • Shard count: Avoid both oversized shards and excessive shard counts because every searched shard adds coordination and result-merging overhead.

  • Aliases: Query through aliases so indices can be reindexed, rolled over, or replaced without changing application requests.

  • Routing: Custom routing can colocate related documents, but every operation requiring that locality must use the same routing value; otherwise it can create hot shards.

  • Lifecycle management: Elasticsearch uses ILM, while OpenSearch uses ISM. Both support data streams, although policy-management details differ.

  • Segment maintenance: Do not routinely use force_merge; normal merging is safer for active, write-heavy indices.

Engine Differences

Concern Elasticsearch 8.x OpenSearch 2.x/3.x
Lifecycle Data streams with ILM Data streams with ISM
Vector mapping dense_vector knn_vector with the k-NN plugin
Vector query knn query Plugin-provided knn query

Quantization and other vector optimizations depend on the exact release, vector engine, and index settings. Verify supported options before using them.

Code Example

Create an index with explicit mappings before ingesting products:

PUT /products-v1
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "category": { "type": "keyword" },
      "price": {
        "type": "scaled_float",
        "scaling_factor": 100
      }
    }
  }
}

Use BM25 for relevance and filters for exact conditions:

GET /products-v1/_search
{
  "_source": ["name", "category"],
  "size": 20,
  "track_total_hits": false,
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "name": "wireless headphones"
          }
        }
      ],
      "filter": [
        {
          "term": { "category": "audio" }
        },
        {
          "range": {
            "price": {
              "gte": 5000,
              "lte": 20000
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "categories": {
      "terms": {
        "field": "category",
        "size": 10
      }
    }
  }
}

To diagnose a slow query, temporarily add "profile": true to the search body and inspect the response’s profile object. Profiling adds overhead and should not remain enabled continuously.

Production Operations and Best Practices

  • Monitor p95/p99 latency, CPU, JVM garbage collection, heap pressure, disk watermarks, rejected tasks, shard balance, cache behavior, and slow logs.

  • Treat a red cluster as an availability incident. A yellow cluster normally indicates an unassigned replica and requires investigation.

  • Use the Bulk API, monitor write queues, and account for refresh, merge, and disk pressure when ingestion affects search latency.

  • Enable TLS, apply least-privilege index permissions, restrict administrative access, and audit privileged activity.

  • Take regular snapshots and test restores. Rolling upgrades must follow the engine’s supported procedure while monitoring allocation, health, and latency.

  • Profile representative slow queries before changing settings; test mapping changes with a reindex, verify shard balance, and validate changes under representative concurrency.

66

What are slow logs in Elasticsearch and OpenSearch?

Slow logs are node-local diagnostic records of indexing, query, or fetch operations that exceed configured duration thresholds. They help identify unusually expensive work and latency spikes, but an entry is not automatically an error and does not replace end-to-end latency monitoring.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 3.x

Types of Slow Logs

Slow log What it measures Common causes
Indexing slow log Time spent executing an indexing operation Hot shards, disk pressure, large documents, mapping overhead, frequent refreshes
Query slow log Time spent in the search query phase Expensive filters, high-cardinality aggregations, wildcard queries, excessive shard counts
Fetch slow log Time spent retrieving and transferring search hits Large _source documents, many hits, slow disks or networks

Query and fetch thresholds are independent: a query can execute quickly while fetching and transferring many large documents is slow.

How They Work

  • Disabled by default: Elasticsearch and OpenSearch do not record slow operations until thresholds are configured.

  • Severity thresholds: Each operation type supports warn, info, and debug thresholds. Thresholds are independent, so enabling a more verbose level does not disable a less verbose one.

  • Recorded after completion: A long-running operation that has not finished may not yet have a slow-log entry. The entry describes work that completed above the threshold.

  • Node-local output: Each node writes records for work executed on that node. Production collection must cover every node rather than only coordinating nodes.

  • Structured metadata: Entries are JSON records containing timing and operation metadata. Depending on the product and operation, they may include index names, usernames, client-origin information, and shard details; they do not necessarily contain the full query body or indexed documents.

  • No automatic remediation: Logging does not cancel, retry, or reroute work. Diagnosis requires correlating entries with cluster metrics, shard state, system telemetry, and application request logs.

Product-Specific Configuration and Output

Product Settings Output
Elasticsearch 8.x Static node settings use logging.slowlog.threshold.indexing.* and logging.slowlog.threshold.{query,fetch}.*. Dynamic cluster settings include logging.slowlog.threshold.indexing.*, search.slowlog.threshold.query.*, and search.slowlog.threshold.fetch.*. Indexing: logs/slowlog.json; search: logs/slowlog_search.json, relative to path.logs.
OpenSearch 3.x Indexing thresholds use index.slowlog.threshold.indexing.* and can be managed per index. Search query and fetch thresholds use plugins.query.slowlog.threshold.{query,fetch}.* and can be changed through dynamic cluster settings. The search slow-log component/plugin must be present on every node. Indexing and search entries are written to logs/opensearch-slowlog.json under OPENSEARCH_HOME.

Prefer dynamic settings for temporary investigation or controlled tuning. Changing a static value in elasticsearch.yml or opensearch.yml normally requires a node restart, so use a rolling restart when the change must be made statically.

Production Practices

  • Start conservatively: Begin with warn and a realistic threshold. debug logging on a busy cluster can add disk I/O, ingestion overhead, and log-storage pressure.

  • Correlate with telemetry: Use /_nodes/stats/indices/search, /_cat/indices, /_cluster/health, JVM and garbage-collection metrics, disk latency, and CPU saturation.

  • Investigate shard design: Oversized indices, excessive shard counts, hot shards, and poorly selective queries commonly produce recurring slow operations.

  • Use profiling selectively: The Profile API exposes per-shard query and fetch details for a representative request, but profiling itself can add overhead and should not be enabled continuously.

  • Protect sensitive metadata: Restrict access to slow-log files and central log pipelines. Treat index names, usernames, and client-origin data as potentially sensitive, while not assuming query bodies or documents are present.

  • Manage retention: Slow-log files can consume significant disk space and may not be rotated like ordinary service logs. Configure collection, rotation, retention, and capacity alerts.

  • Use request correlation: Because an entry may not contain the complete request, preserve application request IDs and timestamps when possible so a slow operation can be matched to its client request and response.

67

What are cross-cluster search and replication in Elasticsearch?

Cross-cluster search (CCS) executes one distributed search across multiple Elasticsearch clusters without moving the indexed data. Cross-cluster replication (CCR) continuously copies an index from a leader cluster to a follower cluster, providing cross-cluster availability and recovery capabilities.

Read full answer

Applies toElasticsearch 8.x; OpenSearch 2.x/3.x with feature-specific differences

Architecture and Terminology

  • A cluster contains nodes, indices, and shards. Each primary shard has a Lucene index and can have replicas within the same cluster.

  • CCS sends a search to configured remote clusters and merges their responses with the local results.

  • CCR maintains a follower index that mirrors a leader index in another cluster. The follower has its own primary shards and replicas.

  • Ordinary primary/replica replication is not CCR: ordinary replicas belong to one cluster, while CCR connects separate clusters.

Cross-Cluster Search

A coordinating node receives the request, fans it out to matching shards in each cluster, and combines hits and aggregation results. Data remains in its original cluster.

  • Remote indices use the form <remote-alias>:<index-pattern>.

  • Mappings, analyzers, and field types must be compatible if scores from different clusters will be compared.

  • Each shard applies its local analyzer and BM25 scoring. Different mappings or analyzer settings can produce inconsistent relevance scores.

  • Aggregations execute on remote shards and are combined centrally, but broad date ranges and high-cardinality aggregations can consume substantial memory and network bandwidth.

  • CCS does not provide snapshot-level consistency. A multi-cluster search can observe slightly different points in time in each cluster.

  • An unavailable remote cluster may cause skipped shards, partial results, or a failed request, depending on remote-cluster settings and request handling.

Cross-Cluster Replication

CCR restores the leader index into the follower and then applies subsequent Lucene operations.

  • The leader retains operation history, including soft-deleted versions, that followers need to catch up.

  • The follower initially performs a remote restore and later continues from replication checkpoints.

  • Writes and deletes occur on the leader. Follower indices are read-only and must not receive independent writes.

  • The follower has its own shard allocation, replicas, and capacity requirements.

  • Index mappings and settings are replicated, but ingest pipelines, ILM policies, index templates, and other cluster-management objects should be managed separately.

  • If a follower falls behind too far to replay retained history, CCR must resynchronize it, potentially through another restore or snapshot-based recovery.

  • Promotion makes a follower writable for disaster recovery, but it is then a separate index with independent lifecycle management.

Concern CCS CCR
Purpose Search several clusters Maintain a follower copy
Data movement Search requests and responses Index data and operations
Consistency Time-varying partial view Follower catches up with leader
Writes No effect on indexed data Only through the leader
Main risk Partial results or score inconsistency Replication lag or failed recovery

OpenSearch Differences

OpenSearch provides cross-cluster search and replication through its own plugins, configuration, privileges, and APIs. Elasticsearch CCS and CCR settings, routing syntax, and security privileges should not be assumed to work unchanged in OpenSearch. Use the documentation matching the deployed OpenSearch release.

Production Operations

  • Monitoring: Track CCS failures, skipped shards, timeouts, and query latency. For CCR, inspect GET /_ccr/stats, follower lag, restore progress, and failed follower tasks.

  • Cluster health: Use GET /_cluster/health and allocation explanations. Red means a primary is unavailable; yellow means replicas are unassigned; green means all expected shards are allocated.

  • Shard sizing: Start with roughly 20–40 GB per shard for many search workloads, then benchmark. Avoid thousands of tiny shards, and use rollover and ILM to keep shard counts controlled.

  • Security: Enable TLS, grant task-specific remote privileges, restrict index patterns, and never use administrative credentials for remote connections.

  • Failure planning: Test network partitions, leader outages, history-retention expiry, follower resynchronization, and promotion. CCR complements backups but does not replace them; retain versioned, tested snapshots.

68

How does vector search work in Elasticsearch with dense_vector and kNN?

Vector search in Elasticsearch stores embeddings in a dense_vector field and uses an HNSW graph to find approximate nearest neighbors with the knn query clause. Results are ranked by the field’s configured similarity function, while exact distance calculation remains available through script_score.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.17–2.19

How It Works

  • A dense_vector field stores a fixed-length numeric embedding. With index: true, Elasticsearch builds HNSW graphs as Lucene segments are flushed and merged.

  • Each shard traverses its graph to find candidates. num_candidates controls the candidate width considered per shard; a coordinating node then produces the requested global top k.

  • Approximate HNSW search trades exactness for lower latency. A script_score query can calculate exhaustive distances, but its cost grows with the number of vectors examined.

  • A filter inside the knn clause is applied before candidate selection, avoiding the recall loss commonly seen when filtering only the final hits. Very selective filters can still increase traversal work.

  • The query vector must contain exactly dims values. Newer Elasticsearch 8.x releases also provide quantized dense_vector index_options; these should be evaluated against recall and relevance tests before use.

Similarity Metrics

Elasticsearch metric Requirement and behavior
cosine Uses cosine similarity and handles vector normalization.
dot_product Faster, but indexed and query vectors must have unit length.
l2_norm Uses Euclidean distance and is suitable for image or feature vectors.
max_inner_product Supports MIPS-style retrieval where vector magnitude matters.

OpenSearch Differences

OpenSearch 2.17–2.19 does not use Elasticsearch’s dense_vector field. It uses knn_vector with a dimension and a method containing fields such as name, space_type, engine, and engine-specific parameters.

nmslib remains available in 2.x but has been deprecated since 2.12; new deployments should use supported engines such as faiss or faiss_lucene, while OpenSearch 2.19 adds the lucene engine. OpenSearch’s approximate-query syntax uses a vector array rather than Elasticsearch’s query_vector, and exact script_score search is also available. The knn_vector field supports quantization from OpenSearch 2.17 onward.

Shard Sizing and Cluster Health

  • Raw float vectors require approximately dims × 4 bytes per document before indexes and graph overhead. Ten million 384-dimensional vectors consume about 14.3 GiB per complete float copy before additional storage and memory costs.

  • A 10–50 GiB shard is a common starting range for conventional search indices, but vector workloads must be benchmarked. Prefer avoiding many tiny shards, while retaining enough shards for parallelism, capacity, and failure isolation.

  • HNSW construction and searching are CPU- and memory-intensive. Monitor JVM pressure, indexing throughput, merge activity, and query latency with GET /_nodes/stats/jvm,indices,thread_pool, GET /_nodes/stats/indexing_pressure, GET /_cluster/health, and GET /_cat/shards.

  • One replica creates another full shard copy, so total cluster storage and indexing work roughly double; replicas improve availability and query distribution.

Code Example

The following runnable toy example uses three dimensions so the mapping and query agree. A production mapping must use the embedding model’s exact output dimension.

PUT /products
{
  "mappings": {
    "properties": {
      "category": {
        "type": "keyword"
      },
      "embedding": {
        "type": "dense_vector",
        "dims": 3,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}
GET /products/_search
{
  "knn": {
    "field": "embedding",
    "query_vector": [0.12, -0.04, 0.31],
    "k": 10,
    "num_candidates": 100,
    "filter": {
      "term": {
        "category": "books"
      }
    }
  }
}

Best Practices and Pitfalls

  • Use dot_product only when both indexed and query vectors are unit length; otherwise use cosine.

  • Set num_candidates explicitly and tune it against a labeled recall set. Larger values generally improve recall but increase latency and resource consumption.

  • Prefer the knn pre-filter for selective constraints, and measure its performance rather than assuming it is always cheaper.

  • Changing dims or similarity requires creating a new index and reindexing. Use aliases for controlled model or mapping cutovers.

  • Combine lexical BM25 retrieval with vector retrieval using Elasticsearch’s rank reciprocal rank fusion for hybrid search where appropriate.

  • Snapshot vector indices, test restores, throttle ingestion during heap pressure, and protect search APIs with TLS, authentication, authorization, and audit logging because queries and embeddings may expose sensitive data.

69

How do you combine BM25 and vector search in Elasticsearch 8.x?

Use hybrid search to combine Elasticsearch’s BM25 lexical ranking with approximate nearest-neighbor (kNN) vector ranking. From 8.16 the standard way to fuse both result lists is the reciprocal rank fusion (RRF) retriever, which merges rankings instead of adding scores on incompatible scales.

Read full answer

Applies toElasticsearch 8.4–8.18 (top-level `knn` search option since 8.4, `knn` query clause since 8.12; RRF retriever since 8.16, technical preview in 8.16–8.17 and generally available in 8.18); OpenSearch 2.10–3.x differences noted

How Hybrid Search Works

  • BM25 branch: Searches the Lucene inverted index with match, term or bool queries. Text is processed by mapping analyzers, and BM25 ranks documents by term frequency, inverse document frequency and field-length normalization.

  • Vector branch: Uses a dense_vector field backed by an approximate nearest-neighbor index, normally HNSW. num_candidates controls how many HNSW candidates are examined per shard, k how many neighbors are returned.

  • Fusion layer: RRF scores a document from its rank in each list, summing 1/(rank + rank_constant) over the retrievers (rank_constant default 60, rank_window_size default 100). It is more reliable than summing BM25 and cosine scores because BM25 is unbounded and not calibrated against vector scores.

  • Execution model: Each branch runs across the index’s shards, RRF is scored per shard, and the coordinating node merges the fused top lists.

Before 8.16 there is no RRF retriever: one search can carry both query and the top-level knn option (since 8.4), which sums the BM25 and vector scores (optionally weighted with boost). That needs score normalization and tuning; prefer RRF from 8.16.

Query and Ranking Choices

  • Use BM25 for identifiers, product codes, names and phrases where lexical precision matters.

  • Use vector search for paraphrases, semantic similarity and multilingual content.

  • Apply security and tenant filters inside both retrievers; filtering only the BM25 branch lets other tenants’ documents influence vector rankings.

  • Tune num_candidates, k and rank_window_size together: higher values improve recall but cost latency and heap.

  • For explicit weighted fusion, use the linear retriever (8.18+, per-retriever weights with optional min-max normalization) instead of RRF.

  • Validate against a labeled evaluation set rather than assuming either method wins.

Code Example

The following creates a three-dimensional vector mapping. A production index should use the actual model’s embedding dimensions.

PUT products
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "standard"
      },
      "body": {
        "type": "text",
        "analyzer": "standard"
      },
      "tenant_id": {
        "type": "keyword"
      },
      "embedding": {
        "type": "dense_vector",
        "dims": 3,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

The RRF request combines BM25 and vector retrieval while enforcing the tenant filter on both paths:

GET products/_search
{
  "size": 20,
  "retriever": {
    "rrf": {
      "retrievers": [
        {
          "standard": {
            "query": {
              "bool": {
                "must": [
                  {
                    "multi_match": {
                      "query": "wireless headphones",
                      "fields": ["title^3", "body"]
                    }
                  }
                ],
                "filter": [
                  {
                    "term": {
                      "tenant_id": "acme"
                    }
                  }
                ]
              }
            }
          }
        },
        {
          "knn": {
            "field": "embedding",
            "query_vector": [0.12, -0.4, 0.87],
            "k": 50,
            "num_candidates": 200,
            "filter": {
              "term": {
                "tenant_id": "acme"
              }
            }
          }
        }
      ],
      "rank_window_size": 100
    }
  }
}

title^3 gives BM25 title matches more importance. With similarity: cosine Elasticsearch normalizes the vectors itself at index time, so no manual normalization is needed; that requirement applies to dot_product, which expects unit-length vectors.

Production Operations

  • Shard sizing: Aim for roughly 20–50 GB per shard for conventional search indexes, while also accounting for vector memory and HNSW graph size. Measure ingestion rate, merge pressure and segment count before choosing shard counts.

  • Monitoring: Track /_cluster/health, /_cat/shards, rejected search tasks, search latency, CPU, heap pressure, garbage collection, disk watermarks and merge duration.

  • Reliability: Use replicas for availability and query distribution, ILM for retention and rollover, and snapshot repositories for recovery; replicas do not replace backups.

  • Model lifecycle: Record the embedding model and version with each document and re-index when dimensions or preprocessing change; mixing incompatible vectors silently degrades relevance.

  • Upgrades: Follow the rolling-upgrade procedure, monitor recovery and cluster state, and avoid index or shard-limit changes during a rollout.

  • Security: Enable TLS and authentication, grant least-privilege roles, protect API keys, and keep the cluster on a private network.

  • Common failures: Disk-watermark blocks, rejected tasks, oversized mappings, too-small num_candidates, stale embeddings and mismatched filters between retrievers.

OpenSearch Differences

OpenSearch uses a knn_vector mapping and its own knn query syntax. Since OpenSearch 2.10 its hybrid query with a normalization search pipeline can combine lexical and vector queries, and 3.x keeps the same general use case. OpenSearch commonly applies explicit normalization and score-processing stages for weighted fusion, so Elasticsearch’s retriever.rrf request should not be copied directly. It also exposes different vector engines (Lucene, FAISS, nmslib) whose tuning affects recall and latency.

71

How do Elasticsearch, Solr, Meilisearch, and Typesense compare?

Elasticsearch is the strongest fit for distributed search and analytics, while Solr offers comparable Lucene-based capability with a different operating model. Meilisearch and Typesense favor simpler deployment, typo-tolerant application search, and faster product-search integration.

Read full answer

Applies toElasticsearch 8.x, OpenSearch 2.x, Apache Solr 9.x, Meilisearch 1.6+, Typesense 28.x–30.x

Positioning and Architecture

Product Architecture Best fit
Elasticsearch Lucene-based distributed cluster; shards on data nodes; replicas Complex search, analytics, and vectors
OpenSearch Elasticsearch-derived fork with separate plugins and releases Open-source search/analytics and AWS integrations
Apache Solr Lucene plus SolrCloud collections, shards, and replicas Mature enterprise search and faceting
Meilisearch Purpose-built inverted index, usually one service instance Easy, fast application search
Typesense In-memory inverted index with multi-node document replication Low-latency product search

Elasticsearch and OpenSearch are separate products after the fork; their APIs overlap, but plugins, releases, security tooling, and managed services differ. Solr shares Lucene but has its own administration and query APIs. Meilisearch and Typesense do not use Lucene and are not drop-in replacements.

In Lucene systems, a shard contains immutable segments that are periodically merged. Mappings define field types and analyzers control tokenization. Elasticsearch and OpenSearch both provide ingest pipelines, but lifecycle management differs: Elasticsearch ILM versus OpenSearch Index State Management. Elasticsearch nodes can coordinate searches by default; dedicated coordinating nodes are unnecessary.

Search and Relevance

Capability Elasticsearch/OpenSearch Solr Meilisearch Typesense
Default relevance BM25 BM25 Custom ranking rules Custom ranking rules
Typo tolerance Fuzzy queries or analysis components Fuzzy queries or analysis components Built in and prominent Built in and prominent
Facets and aggregations Very extensive Extensive Limited Strong faceting; limited ranges
Vector search Elasticsearch: dense_vector and k-NN queries; OpenSearch: knn_vector and k-NN plugin Available, but version-dependent Experimental user-supplied vectors Available, with different constraints

Elasticsearch and OpenSearch provide match, term, bool, and filter clauses in their Query DSL. match analyzes text, whereas term does not and is normally used for keyword fields. BM25 considers term frequency, inverse document frequency, and field-length normalization. Meilisearch and Typesense deliberately simplify relevance, filtering, and aggregation APIs for storefronts, documentation, and directories.

Scaling and Operational Trade-offs

  • Elasticsearch/OpenSearch shard sizing: Use approximately 10–50 GB per primary shard as an initial benchmark, then measure. Small shards consume heap and cluster-state resources; oversized shards slow recovery and rebalancing.

  • Replication: Replicas improve availability and can share search work, although shard selection and routing determine where queries execute. They also require additional disk and write capacity.

  • Cluster health: Green means all primaries and replicas are allocated. Yellow means primaries are allocated but at least one replica is missing. Red means one or more primaries are unavailable. Check allocation explanations before trusting automatic recovery.

  • SolrCloud: Distributed collections provide sharding and replicas, but placement, balancing, backups, and upgrades require specialized operational expertise.

  • Meilisearch and Typesense: They avoid large sharded coordination layers. Typesense provides asynchronous multi-node replication; Meilisearch generally requires multiple instances, cloud HA, or an external replication design.

Production Operations, Security, and Failure Modes

  • Monitoring: Monitor Elasticsearch/OpenSearch cluster and node health, JVM heap and garbage collection, disk latency and watermarks, allocation, indexing and search latency, rejected tasks, and snapshot failures. Track equivalent JVM, disk, query, index, and replication metrics for Solr, Meilisearch, and Typesense.

  • Backups: Use and test Elasticsearch/OpenSearch repository snapshots, Solr backup/restore capabilities and SolrCloud replication, Meilisearch dumps, and Typesense snapshots or replication. Validate recovery time and recovery point objectives.

  • Security: Encrypt client and inter-node traffic with TLS where supported, or terminate it at a trusted proxy. Use role-based authorization for clustered engines, API keys or equivalent controls for Meilisearch and Typesense, restrict administrative endpoints, encrypt backups, and rotate credentials.

  • Failure modes and upgrades: Disk pressure, JVM pressure, corrupt shards, mapping conflicts, and allocation conflicts can degrade or interrupt service. Upgrade one node at a time where rolling upgrades are supported, observe the product’s mixed-version compatibility window, back up before major upgrades, and monitor segment merges and reindexing.

When to Choose Each

Product Choose it when
Elasticsearch Complex relevance, distributed ingestion, extensive analytics, ILM, data streams, and native vector search are required
OpenSearch An Elasticsearch-like model and the OpenSearch plugin or managed-service ecosystem are priorities
Solr Enterprise search, faceting, custom indexers, and existing SolrCloud expertise justify its operational model
Meilisearch Fast integration, minimal administration, and excellent typo tolerance outweigh advanced analytics and distributed shard control
Typesense Low-latency product search, strong filtering, faceting, and simple multi-node replication fit the workload
72

What are the most common Elasticsearch anti-patterns in production?

The most common Elasticsearch anti-patterns involve poor shard sizing, incorrect mappings, expensive queries, inefficient ingestion, and inadequate operational controls. These mistakes can reduce search performance, consume excessive heap memory, weaken resilience, and make recovery or upgrades unnecessarily difficult.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Architecture and Sharding

  • Undersized shards: Every shard is a Lucene index with per-shard segment and filesystem overhead; shards on the same node share that node’s JVM heap. Thousands of tiny shards increase cluster-state and scheduling overhead.

  • Oversized shards: A very large shard slows recovery, reindexing, and shard relocation. A common starting point is approximately 10–50 GB per shard, but workload, hardware, retention, and query patterns must determine the final size.

  • Arbitrary shard counts: The primary shard count is fixed when an index is created, while the replica count can be changed. Use enough primary shards to distribute work across data nodes, then validate the design with representative indexing and search tests.

  • No replicas: A production index should normally have at least one replica for availability and read capacity. A zero-replica index has no shard-level redundancy, so losing a primary makes the affected data unavailable until recovery.

  • Unbalanced allocation: Primary and replica copies of the same shard must not be allocated to the same node. Spread replicas across failure domains, such as availability zones.

  • Ignoring Lucene internals: Indexing creates Lucene segments, refresh makes those segments searchable, and merges later consolidate them. Excessive refresh frequency creates many small segments, while frequent updates increase merge pressure.

A shard is Elasticsearch’s unit of index data partitioning. Searches fan out to relevant shards, and a coordinating node merges their results and scores.

Data Modeling and Search

  • Using text for every field: Use text for analyzed human-readable content and keyword for exact values, identifiers, filters, and aggregations.

  • Confusing match and term: match analyzes the query using the field’s analyzer; term performs an exact lookup without analysis. Use term for exact keyword values, not arbitrary search phrases.

  • Overly broad dynamic mappings: Uncontrolled field names can cause a mapping explosion and excessive memory usage. Use explicit mappings, dynamic templates, or controlled ingestion schemas.

  • Misusing object arrays: Object arrays are flattened, which can create matches between unrelated nested objects. Use a nested mapping when array elements must be queried independently.

  • Disabling useful storage: Disabling _source prevents use of the Update API and source-based partial retrieval and makes reindexing harder. Avoid disabling doc_values for frequently sorted or aggregated fields.

  • Expensive queries: Leading wildcards, broad prefix searches, high-cost scripts, and unbounded date ranges can overwhelm nodes. Put non-scoring conditions in a bool query’s filter context, which avoids scoring and is eligible for request caching.

Aggregations also require care. High-cardinality terms aggregations and cardinality aggregations can consume substantial memory; cardinality is approximate. Use composite aggregations for reliable, paginated aggregation exports.

Ingestion, Lifecycle, and Vectors

  • Indexing one document per request: Use the Bulk API for batches, enable appropriate backoff, and monitor rejected bulk tasks. This reduces HTTP and indexing overhead.

  • Overly frequent refreshes: Refresh controls search visibility, not durability. Increasing it for every event creates many small segments. Bulk indexing with a deliberate refresh interval is usually more efficient.

  • Deleting data with delete-by-query: Repeated broad deletions generate expensive merges and can destabilize the cluster. Use time-based indices, data streams, and lifecycle policies.

  • No retention policy: Elasticsearch uses ILM policies, while OpenSearch commonly uses ISM policies. These APIs are not interchangeable.

  • Ignoring vector-search differences: Elasticsearch 8.x uses dense_vector; OpenSearch uses knn_vector with its k-NN plugin. Use native approximate nearest-neighbor search rather than custom scripts that scan every vector.

Operations, Reliability, and Security

  • Treating yellow as harmless: Green means all primaries and replicas are allocated; yellow means all primaries are allocated but some replicas are not; red means at least one primary is unavailable.

  • Running without disk headroom: The flood-stage disk watermark can trigger a protective write block. Monitor filesystem usage and keep capacity for merges, translogs, and temporary files.

  • Skipping backups: Configure a remote snapshot repository, monitor snapshot failures, and periodically test restoration. A snapshot that has never been restored is not a proven recovery plan.

  • Breaking rolling upgrades: Upgrade supported node versions one at a time, follow the product’s compatibility requirements, and check that replicas are allocated before proceeding.

  • Exposing unsecured endpoints: Enable TLS, RBAC, least-privilege API keys or roles, audit logging, and network restrictions. Never expose port 9200 publicly without authentication.

  • Not monitoring the right signals: Track cluster health, JVM heap and GC pauses, disk latency and space, search latency, indexing rates, rejected tasks, segment counts, and unassigned shards.

Production Best Practices

  • Load-test index creation, updates, searches, aggregations, merges, and recovery—not only steady-state queries.

  • Prefer data streams and time-based rollover for append-only workloads.

  • Reindex when a mapping requires structural changes; most mapping fields cannot be changed safely in place.

  • Use GET /_cluster/health?level=indices, GET /_cat/shards?v, and GET /_nodes/stats/jvm as starting points for operational checks.

  • Define alerts for red clusters, persistent yellow clusters, unassigned shards, disk pressure, heap pressure, rejected tasks, and failed snapshots.

73

How do I search products by name, filter by price range, and aggregate by brand in Elasticsearch?

Product search uses an Elasticsearch bool query: put the name match in must so BM25 affects relevance, put the price range in filter so it does not affect scoring, and use a terms aggregation on a keyword brand field for the breakdown.

Read full answer

Applies toElasticsearch 8.x (the `knn` query requires 8.12+) and OpenSearch 2.x with the k-NN plugin

How It Works

  • Shard execution: the coordinating node sends the request to one eligible copy of each shard, then reduces shard-level hits and aggregation results.

  • Name matching: match analyzes its input using the field’s analyzer. Unlike term, it can find analyzed text such as wireless headphones.

  • Price filtering: range in filter context does not contribute to _score. gte and lte are inclusive; gt and lt are exclusive.

  • Brand aggregation: terms returns buckets ordered by document count. Each bucket contains doc_count, and the price filter constrains those counts.

  • Relevance: Elasticsearch 8.x uses BM25 by default with k1: 1.2 and b: 0.75. Term frequency and field length affect the ranking of matching name documents.

Required Mapping

Use a keyword field for exact brand aggregation. Aggregating a text field normally fails because fielddata is disabled by default.

PUT /products
{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword", "ignore_above": 256 }
        }
      },
      "brand": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 }
    }
  }
}
  • Price precision: send prices in currency units, such as 50.25; scaled_float stores that value internally as 5025. Use long if more than two decimal places are required.

  • Exact names: name.keyword supports exact matching with term, such as Canon EOS R6.

Code Example

Kibana Dev Tools and OpenSearch Dashboards Dev Tools accept the same REST request:

GET /products/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must": [
        { "match": { "name": "wireless headphones" } }
      ],
      "filter": [
        { "range": { "price": { "gte": 50, "lte": 500 } } }
      ]
    }
  },
  "aggs": {
    "by_brand": {
      "terms": { "field": "brand", "size": 10 },
      "aggs": {
        "avg_price": { "avg": { "field": "price" } },
        "price_ranges": {
          "range": {
            "field": "price",
            "ranges": [
              { "to": 100 },
              { "from": 100, "to": 300 },
              { "from": 300 }
            ]
          }
        }
      }
    }
  }
}

Equivalent elasticsearch-py 8.x request:

import os
from elasticsearch import Elasticsearch

# Use an HTTPS URL outside a trusted development network.
es = Elasticsearch(
    os.environ["ELASTICSEARCH_URL"],
    basic_auth=(
        os.environ["ELASTICSEARCH_USER"],
        os.environ["ELASTICSEARCH_PASSWORD"],
    ),
)

response = es.search(
    index="products",
    query={
        "bool": {
            "must": [{"match": {"name": "wireless headphones"}}],
            "filter": [{"range": {"price": {"gte": 50, "lte": 500}}}],
        }
    },
    size=20,
    aggs={
        "by_brand": {
            "terms": {"field": "brand", "size": 10},
            "aggs": {
                "avg_price": {"avg": {"field": "price"}},
                "price_ranges": {
                    "range": {
                        "field": "price",
                        "ranges": [
                            {"to": 100},
                            {"from": 100, "to": 300},
                            {"from": 300},
                        ],
                    }
                },
            },
        }
    },
)

for hit in response["hits"]["hits"]:
    print(hit["_source"]["name"], hit["_score"])

for bucket in response["aggregations"]["by_brand"]["buckets"]:
    print(bucket["key"], bucket["doc_count"], bucket["avg_price"]["value"])

OpenSearch Differences

  • The shown query, mapping, and aggregation syntax work unchanged in OpenSearch 2.x.

  • took, timed_out, _shards, and hits.total.relation have the same basic response shape.

  • OpenSearch vector search requires the k-NN plugin, with method and engine settings defined in the index mapping. Elasticsearch’s knn query over dense_vector requires 8.12 or later.

  • Use OpenSearch Dashboards Dev Tools for OpenSearch and Kibana Dev Tools for Elasticsearch.

Production Pitfalls and Best Practices

  • Bucket limit: search.max_buckets defaults to 65,536 per search. Use a composite aggregation when paginating over many brand buckets.

  • Shard sizing: target roughly 20–50 GB per primary shard and avoid many tiny shards. Each shard adds heap, segment, and merge overhead.

  • Rollover: use an Elasticsearch data stream with ILM for growing catalogs, normally combining max_age or max_docs with max_primary_shard_size. Use OpenSearch ISM policies for equivalent workflows.

  • Cluster health: yellow means some replicas are unassigned. Causes include missing nodes, disk watermarks, and allocation constraints; investigate the allocation-explanation API rather than assuming one cause.

  • Deep pagination: avoid deep from/size pagination beyond the default 10,000-result window; use search_after with a PIT.

  • Hit counts: set track_total_hits: false when exact totals are unnecessary.

  • Monitoring: watch search latency, JVM GC, segment count, rejected requests, and disk usage using the dashboards and metrics available in the installed release.

  • Security and backups: enable TLS and authentication, grant the application only read on products, and do not expose port 9200 publicly. Use snapshot repositories such as S3 or shared storage and regularly test restores.

74

How do I create an Elasticsearch mapping for blog posts?

An index mapping defines how Elasticsearch stores, analyzes, indexes, and retrieves blog-post fields. A text mapping supports full-text analysis for titles and bodies, while keyword preserves exact values such as tags for filtering and aggregation.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x/3.x

Mapping Design

Field Mapping Purpose
title text with a keyword subfield Full-text search plus exact-title filtering or sorting for values within the subfield’s ignore_above limit
body text Full-text search through analyzed terms
tags keyword Exact filtering, aggregations, and multi-value storage
publish_date date Range queries, sorting, filtering, and date aggregation
  • Analysis: title and body are analyzed using the standard analyzer by default, producing lowercase tokens suitable for match queries.

  • Exact matching: tags is not analyzed, so a term query for devops matches only that exact tag.

  • Date format: strict_date_optional_time||epoch_millis accepts values such as 2025-01-15T09:30:00Z and Unix epoch milliseconds.

  • Strict fields: dynamic: strict rejects documents containing unmapped fields, helping prevent accidental schema changes.

Code Example

The following requests can be run directly in Kibana Dev Tools or OpenSearch Dashboards Dev Tools:

PUT /blog-posts
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "body": {
        "type": "text"
      },
      "tags": {
        "type": "keyword"
      },
      "publish_date": {
        "type": "date",
        "format": "strict_date_optional_time||epoch_millis"
      }
    }
  }
}

Index a document containing multiple tags:

POST /blog-posts/_doc
{
  "title": "Running Elasticsearch in Production",
  "body": "Guidance for shard sizing, monitoring, backups, and upgrades.",
  "tags": ["elasticsearch", "devops", "sre"],
  "publish_date": "2025-01-15T09:30:00Z"
}

A useful search combines full-text matching with exact filters:

GET /blog-posts/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "body": "production monitoring"
          }
        }
      ],
      "filter": [
        {
          "term": {
            "tags": "elasticsearch"
          }
        },
        {
          "range": {
            "publish_date": {
              "gte": "2025-01-01T00:00:00Z"
            }
          }
        }
      ]
    }
  }
}

How the Fields Are Indexed

  • Inverted index: Elasticsearch indexes each analyzed term and records which documents contain it. A match query analyzes its input, while a term query searches the indexed value without analysis.

  • Lucene segments: Documents are written into Lucene segments, which are periodically refreshed and merged. Newly indexed documents become searchable after a refresh, so search is near real-time rather than immediately consistent.

  • Keyword precision: Tags remain exact and can be used with terms queries or bucket aggregations. The title.keyword subfield supports exact-title operations for values within its ignore_above limit; longer titles are not indexed in that subfield.

  • Mapping changes: Existing field types generally cannot be changed in place. A mapping correction usually requires creating a new index and using the reindex API.

Production Practices

  • Use index templates: Define mappings in a versioned template so newly created blog indices receive the same schema.

  • Size shards carefully: Aim for indices large enough to avoid excessive shard counts but generally within a manageable size—often tens of gigabytes per shard after benchmarking. Avoid creating many small indices.

  • Use replicas for availability: Replicas provide redundancy and can serve searches, although they add storage and write overhead. A single-node development cluster normally uses zero replicas; production clusters commonly use at least one.

  • Manage lifecycle policies: Use Elasticsearch ILM for rollover and retention policies and Elasticsearch SLM for scheduled snapshots. OpenSearch ISM can perform rollover, deletion, and snapshot actions; OpenSearch Snapshot Management is also commonly used for recurring cluster backups.

  • Monitor capacity and behavior: Track cluster health, rejected requests, JVM heap, disk usage and watermarks, shard sizes, indexing latency, search latency, and segment count.

  • Protect data: Enable TLS and role-based access control, avoid exposing the cluster directly, and test snapshot restoration rather than relying only on replication.

Elasticsearch and OpenSearch Differences

The mapping and indexing syntax shown here works in both Elasticsearch 8.x and OpenSearch 2.x/3.x. Lifecycle automation differs: Elasticsearch uses ILM for index lifecycle management and SLM for snapshot policies, while OpenSearch provides ISM and Snapshot Management. Security configuration, plugins, and some advanced APIs differ, but the text, keyword, and date mappings remain compatible.

75

How do you design e-commerce product search with Elasticsearch?

E-commerce product search is typically built as a dedicated Elasticsearch index (or set of indices) fronted by a search service, combining full-text relevance, faceted filtering, and business-aware ranking over a denormalized product document.

Read full answer

Applies toElasticsearch 8.x (OpenSearch 2.x noted where it differs).

Architecture and Index Strategy

  • Use a single index per locale/language (e.g. products-en, products-de) behind an alias (products-current) so reindexing and cutover are atomic.

  • Keep the document denormalized: one document per sellable SKU/variant, with nested arrays for attributes, so a single query returns everything the UI needs.

  • Node roles: dedicated master-eligible nodes (3, odd count), data nodes (hot tier), and coordinating nodes behind a load balancer. In large production clusters, avoid co-locating master and data roles.

  • Replicas: at least 1 replica for HA; increase to 2 for read-heavy traffic.

Mappings and Analyzers

Define an explicit mapping and disable dynamic field creation ("dynamic": "strict") to prevent mapping explosions from arbitrary attributes.

PUT /products-en
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "analysis": {
      "tokenizer": {
        "autocomplete_tokenizer": {
          "type": "edge_ngram",
          "min_gram": 2,
          "max_gram": 20
        }
      },
      "filter": {
        "product_synonyms": {
          "type": "synonym_graph",
          "synonyms": ["iphone, apple phone"]
        }
      },
      "analyzer": {
        "product_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding", "product_synonyms"]
        },
        "autocomplete": {
          "tokenizer": "autocomplete_tokenizer",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "title": { "type": "text", "analyzer": "product_analyzer", "fields": { "keyword": { "type": "keyword" } } },
      "description": { "type": "text", "analyzer": "product_analyzer" },
      "brand": { "type": "keyword" },
      "category": { "type": "keyword" },
      "price": { "type": "scaled_float", "scaling_factor": 100 },
      "rating": { "type": "float" },
      "in_stock": { "type": "boolean" },
      "popularity": { "type": "integer" },
      "embedding": { "type": "dense_vector", "dims": 384, "index": true, "similarity": "cosine" }
    }
  }
}
  • text fields use analyzers for full-text; keyword sub-fields support exact filters, sorting and aggregations.

  • Use synonym_graph for brand/model synonyms (“iphone” ↔ “apple phone”).

  • edge_ngram tokenizer powers typeahead/autocomplete.

Query DSL and Relevance

Use bool with must for text relevance and filter for exact constraints. Filters run in filter context: cached, no scoring, faster.

GET /products-current/_search
{
  "query": {
    "function_score": {
      "query": {
        "bool": {
          "must": [
            { "multi_match": { "query": "wireless headphones", "fields": ["title^3", "description"], "type": "best_fields" } }
          ],
          "filter": [
            { "term": { "category": "audio" } },
            { "term": { "in_stock": true } },
            { "range": { "price": { "gte": 50, "lte": 300 } } }
          ],
          "should": [
            { "term": { "brand": "Sony" } }
          ]
        }
      },
      "functions": [
        { "field_value_factor": { "field": "popularity", "factor": 1.2, "modifier": "log1p" } },
        { "gauss": { "rating": { "origin": 5, "scale": 1, "decay": 0.5 } } }
      ],
      "boost_mode": "multiply"
    }
  }
}
  • BM25 is the default scorer; tune k1/b only if needed.

  • Boost title over description; use function_score for business signals (popularity, rating, recency).

  • For typo tolerance, add fuzziness: "AUTO" to match queries.

Facets and Aggregations

"aggs": {
  "brands": { "terms": { "field": "brand", "size": 20 } },
  "price_ranges": { "range": { "field": "price", "ranges": [{ "to": 50 }, { "from": 50, "to": 150 }, { "from": 150 }] } }
}
  • Use terms for brand/category facets, range for price buckets.

  • For nested variants (size/color), use nested aggregations.

Ingestion and Lifecycle

  • Ingest via the Bulk API from your catalog service; use an ingest pipeline for enrichment (e.g. computing embedding).

  • Use ILM with hot-warm-cold tiers: hot for active catalog, warm for older, cold/frozen for archive. Rollover by size/age.

  • Reindex with reindex + alias swap for mapping changes; never update mappings in place for breaking changes.

Vector and Hybrid Search

  • Store embeddings in dense_vector; run knn search for semantic matching.

  • In OpenSearch, use knn_vector instead of dense_vector.

  • Combine lexical + semantic with hybrid search (RRF, available in Elasticsearch 8.8+) to handle both exact SKU queries and natural-language queries (“comfortable running shoes for flat feet”).

Scaling, Shard Sizing and Health

  • Target 10–50 GB per shard; shard count = (data size × (1 + replicas)) / target. Avoid thousands of tiny shards.

  • Monitor cluster health (green/yellow/red), unassigned shards, JVM heap, and slow logs.

  • Use searchable snapshots for cold data; async search for heavy aggregations.

Security and Operations

  • Enable TLS on transport and HTTP; use RBAC with least-privilege roles; prefer API keys over basic auth for services.

  • Back up with snapshot/restore to S3/GCS on a schedule; test restores.

  • Upgrades: rolling upgrades within major version; full reindex across major versions.

Best Practices and Pitfalls

  • Don’t over-shard; shard count is fixed at index creation (unless split).

  • Avoid wildcard/regexp on large text fields; use match_phrase or n-grams instead.

  • Paginate with search_after + PIT, not deep from/size.

  • Cache filter-heavy queries; monitor cache hit rates.

  • OpenSearch differs mainly in plugin ecosystem, security plugin (OpenSearch Security), and some API names; core Query DSL and mappings are largely compatible.

Code Example

POST /_bulk
{ "index": { "_index": "products-en", "_id": "sku-123" } }
{ "title": "Sony WH-1000XM5 Wireless Headphones", "brand": "Sony", "category": "audio", "price": 349.99, "in_stock": true, "popularity": 9500, "rating": 4.7 }
76

How do you design a centralized logging platform with Elasticsearch?

A centralized logging platform should use Elastic Agent, Filebeat, or Fluent Bit to collect logs, Elasticsearch data streams for durable storage, ILM for lifecycle management, and Kibana or OpenSearch Dashboards for search and visualization. Production success depends on deliberate mappings, controlled shard growth, backpressure, security, and tested recovery procedures.

Read full answer

Applies toElasticsearch 8.x (8.12 or later for the k-NN API) and OpenSearch 2.x

Architecture and Data Model

  • Topology: Run at least three dedicated master-eligible nodes for production quorum, plus coordinating/ingest nodes and data nodes organized into hot, warm, and cold tiers. Keep Kibana outside the data tier.

  • Storage: Send append-only logs to data streams. Their hidden Lucene backing indices are named like .ds-logs-<name>-000001; rollover creates another backing index.

  • Shards: Each primary shard is an inverted index. Start with one primary and one replica per backing index, then adjust from measured throughput and retention.

  • Mappings: Map @timestamp as date, identifiers such as service and level as keyword, and message as text. Use controlled dynamic templates and monitor index.mapping.total_fields.limit.

Ingestion, Search, and Analytics

  • Pipeline: Send events through Logstash or Elasticsearch ingest processors such as grok, dissect, date, and geoip. Use the bulk API in roughly 5–15 MB batches, retry 429 responses with exponential backoff, and monitor ingest lag.

  • Analysis: The standard analyzer lowercases and splits text; keyword preserves exact values. Use custom analyzers only for known normalization requirements.

  • Lucene: Searches see immutable segments refreshed near real-time, normally every second. Background merges control segment count; do not routinely force-merge indexes.

  • Queries: match analyzes text, term matches exact keyword values, and bool.filter clauses do not contribute to scoring. BM25 is the default relevance model. Use aggregations for service, error-rate, and time-series dashboards.

  • Vectors: Elasticsearch 8.x uses an indexed dense_vector mapping and the top-level knn query. OpenSearch uses knn_vector with its k-NN plugin. Evaluate recall, latency, and hybrid reranking rather than assuming vector search replaces lexical search.

Lifecycle, Scaling, and OpenSearch

  • Lifecycle: Roll hot indices over by size or age. Reduce replicas and priority in warm, move older data to a searchable snapshot in cold after registering and testing a snapshot repository, and delete it at the retention boundary.

  • Sizing: Target roughly 10–50 GB per primary shard and keep shard counts proportional to throughput and retention. Avoid thousands of tiny shards because each consumes heap and cluster-state resources.

  • OpenSearch: ISM uses _plugins/_ism/policies and a state/action schema rather than Elasticsearch’s phase schema. Data streams exist, but policies, plugins, security, and vector APIs must be validated separately for OpenSearch.

Security and Operations

  • Security: Enable TLS for HTTP and transport, use least-privilege RBAC and scoped API keys, encrypt disks where required, and enable audit logging. Permit clients to reach ingest or coordinating nodes rather than every data node.

  • Monitoring: Track cluster health, unassigned shards, JVM heap and GC, disk watermarks, indexing and search latency, bulk rejections, queue depth, snapshot failures, and capacity trends.

  • Failure modes: A red cluster has an unassigned primary; yellow usually means an unassigned replica. Flood-stage disk watermarks block writes. Treat repeated mapping explosions and sustained 429 responses as capacity or backpressure incidents.

  • Recovery: Schedule snapshots to a supported object-storage repository, test restores, follow the supported rolling-upgrade path, and rehearse node-loss and regional-recovery procedures.

Code Example

PUT /_ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50gb",
            "max_age": "1d"
          },
          "set_priority": { "priority": 100 }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "allocate": { "number_of_replicas": 0 },
          "set_priority": { "priority": 50 }
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {
          "searchable_snapshot": { "snapshot_repository": "logs-snapshots" },
          "set_priority": { "priority": 0 }
        }
      },
      "delete": {
        "min_age": "90d",
        "actions": { "delete": {} }
      }
    }
  }
}

PUT /_index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "data_stream": {},
  "template": {
    "settings": {
      "index.lifecycle.name": "logs-policy",
      "number_of_shards": 1,
      "number_of_replicas": 1
    },
    "mappings": {
      "dynamic_templates": [
        {
          "strings_as_keyword": {
            "match_mapping_type": "string",
            "mapping": { "type": "keyword", "ignore_above": 1024 }
          }
        }
      ],
      "properties": {
        "@timestamp": { "type": "date" },
        "message": { "type": "text" },
        "level": { "type": "keyword" },
        "service": { "type": "keyword" },
        "host": { "type": "keyword" },
        "embedding": {
          "type": "dense_vector",
          "dims": 3,
          "index": true,
          "similarity": "cosine"
        }
      }
    }
  }
}

POST /logs-app/_bulk
{"create":{"_index":"logs-app"}}
{"@timestamp":"2025-01-01T00:00:00Z","level":"error","service":"api","message":"request timed out"}

POST /logs-app/_search
{
  "query": {
    "bool": {
      "must": [
        { "match": { "message": "timed" } }
      ],
      "filter": [
        { "range": { "@timestamp": { "gte": "2020-01-01T00:00:00Z" } } },
        { "term": { "level": "error" } }
      ]
    }
  }
}

Production Best Practices and Pitfalls

  • Version templates and ILM policies; test changes against representative logs before rollout.

  • Use dynamic templates for controlled extensions instead of globally accepting arbitrary fields.

  • Do not combine data streams with shrink actions or routine force-merging.

  • Monitor per-shard size, indexing pressure, rejected writes, and snapshot success, not only cluster color.

  • Keep retention, replica, and searchable-snapshot decisions explicit; each trades cost against availability and recovery time.

77

How do I fix an Elasticsearch low disk space incident?

Treat it as a capacity incident: stop unbounded growth, reclaim or add capacity safely, and restore shard allocation while preserving recoverability. The key control is disk watermarks, which relocate shards and eventually place affected indices in read-only mode.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 2.x

How Disk Pressure Behaves

Elasticsearch and OpenSearch calculate disk pressure from filesystem utilization, including space used by processes other than Elasticsearch.

  • Low watermark: The allocator attempts to move shards away from nodes approaching the limit.

  • High watermark: The allocator avoids allocating additional shards to nodes above the threshold.

  • Flood stage: The affected indices receive a read_only_allow_delete block, preventing normal writes while still allowing deletes.

  • Cluster impact: Replica allocation can fail first, producing a yellow cluster. If primaries cannot recover, the cluster becomes red. Write requests may receive HTTP 429 responses.

Elasticsearch 8.x and OpenSearch 2.x commonly default to low, high, and flood-stage watermarks of 85%, 90%, and 95%, respectively. Verify the effective settings rather than assuming these defaults.

Incident Response

  1. Identify the constrained node and largest indices

Check node allocation, shard distribution, index sizes, and cluster health. Distinguish one full node from a cluster whose total capacity is undersized.

  1. Stop the fastest-growing writer

Temporarily block writes after coordinating with application owners:

http PUT /app-logs-2024.03.14/_settings { "index.blocks.write": true }

This protects the cluster but does not solve the capacity shortage.

  1. Reclaim space or add capacity

  2. Delete obsolete indices only after confirming retention, legal requirements, aliases, and snapshot status.

  3. Roll over high-volume data streams and use Index Lifecycle Management (ILM) in Elasticsearch or Index State Management (ISM) in OpenSearch to enforce retention.

  4. Take a successful snapshot before destructive changes when recovery is required.

  5. Expand the filesystem or add capacity when deletion is unacceptable. Ensure new nodes have appropriate allocation settings and enough headroom for shard movement.

  6. Remove the manual block and verify recovery

A manually applied index.blocks.write block remains until explicitly removed. After reclaiming space, clear it:

http PUT /app-logs-2024.03.14/_settings { "index.blocks.write": false }

The automatic flood-stage read_only_allow_delete block normally clears after disk usage falls below the relevant recovery threshold. Then watch shard relocation and confirm the expected cluster health.

  1. Avoid emergency force merges as the first action

Force merging can cause heavy CPU and I/O, require temporary disk space, and is not a general disk-reclamation mechanism. Use it only for a specific, non-hot index after a snapshot and with sufficient free space.

Monitoring and Shard Sizing

  • Alert before the low watermark, commonly around 75–80% utilization, rather than waiting for writes to be rejected.

  • Monitor filesystem utilization and free bytes, inode usage, I/O latency, indexing rejections, JVM heap, GC time, and allocation movement.

  • Track primary-shard size and request rate. Roughly 10–50 GB per primary shard is a common starting point, adjusted through indexing and query benchmarks.

  • Avoid oversized hot shards and excessive tiny shards because both consume cluster resources.

  • Account for replicas: a 1 TB primary dataset with one replica requires approximately 2 TB before overhead.

  • Set watermarks according to the required headroom. The default flood stage leaves only 5% free, while the example below leaves 10%; use 15–20% for more volatile workloads and configure the flood stage accordingly.

Code Example

The following requests are valid in Kibana Dev Tools and the OpenSearch Dashboards Dev Tools console:

GET /_cat/allocation?v&bytes=gb&s=node,shards,disk.percent,disk.avail

GET /_cat/indices?v&bytes=gb&s=store.size:desc

GET /_cat/shards?v&bytes=gb&s=index,shard,prirep,store

GET /_cluster/health?level=shards

Delete an obsolete index only after confirming its retention status:

DELETE /app-logs-2024.01.01

Configure preventive watermarks rather than raising the flood-stage threshold to conceal pressure:

PUT /_cluster/settings
{
  "persistent": {
    "cluster.routing.allocation.disk.watermark.low": "80%",
    "cluster.routing.allocation.disk.watermark.high": "85%",
    "cluster.routing.allocation.disk.watermark.flood_stage": "90%"
  }
}

Verify recovery after space is reclaimed:

GET /_cluster/health?level=shards

GET /_cat/shards?v&bytes=gb&s=index,shard,prirep,store

Operational Safety and Failure Modes

  • Backups: Snapshots do not reduce live index usage. Store them outside the constrained filesystem and test restoration; restoring into the same cluster can recreate the pressure.

  • Deletion risks: Removing the wrong index or all replicas can make data unavailable. Check retention policies, aliases, data-stream backing indices, and snapshot status first.

  • Security: Restrict index deletion and _cluster/settings access with role-based access control. Elasticsearch 8.x has security enabled by default; secure the OpenSearch Security Plugin appropriately and audit administrative actions.

  • OpenSearch differences: The operational APIs shown are broadly shared, but Elasticsearch uses ILM while OpenSearch commonly uses ISM and its own Security Plugin configuration.

  • Upgrades: Keep nodes within the supported version range and follow the product’s rolling-upgrade procedure, ensuring temporary shard movement has sufficient disk and I/O capacity.

78

How do you migrate to a new Elasticsearch mapping without downtime?

Use a shadow index and atomic alias cutover, with dual writes or a durable change feed to prevent lost updates. An incompatible mapping change generally requires a new index; adding a new field to an existing mapping does not require data migration.

Read full answer

Applies toElasticsearch 8.x and OpenSearch 3.x

Prepare the Shadow Index

  1. Inventory the source

  2. Record aliases, the write index, shard count, settings, analyzers, ingest pipelines, routing, mappings, and lifecycle policy.

  3. Confirm that every client reads and writes through aliases rather than a concrete index name.

  4. Check whether the new mapping is compatible with the existing documents, custom analyzers, dates, keywords, and vectors.

  5. Create the shadow index

  6. Define the complete mapping and compatible settings explicitly.

  7. Start around 10–50 GB per shard, then validate sizing against indexing throughput, query load, and recovery time.

  8. Configure replicas for availability. An index with configured but unassigned replicas is normally yellow, not green.

  9. Define dual-write semantics

  10. Write to both indices using schema-appropriate representations if the old and new mappings are not identical.

  11. Use an application-owned, monotonically increasing revision. Do not compare the internal _version values of two separate indices.

  12. Use external versioning on the destination and make each revision immutable so retries are idempotent.

Backfill and Reconcile

Run _reindex to copy documents and recreate them in the destination. It is not incremental replication, so subsequent changes require dual writes or change-stream processing.

For an initial backfill, use create semantics. This prevents a late stale document from overwriting a newer document already dual-written into the shadow index. Version conflicts caused by concurrent dual writes are expected; other reindex failures must stop cutover.

Record deletes and creates in a durable outbox or change feed. Replay outstanding events after the backfill, especially deletes, because a backfill that started before a deletion could otherwise recreate the deleted document. Do not rely on raw counts while writes continue; compare logical document IDs, revisions, samples, and aggregations at a common reconciliation point.

Cut Over and Roll Back

Before cutover, replay the change feed until its lag is zero and verify that the destination contains the expected revisions. Then update the read and write aliases in one _aliases request.

Continue mirroring writes to the old index during the rollback window if fast rollback is required. Otherwise, rollback requires synchronizing the post-cutover delta back to the old index before reversing the aliases.

If dual writes are impossible, block source writes, recreate the destination, perform a complete _reindex, and then cut over. This is not strict zero-downtime because the write outage lasts for the copy operation.

Code Example

Create the shadow index and perform the initial backfill with create semantics:

PUT /orders-v2
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "order_id": { "type": "keyword" },
      "customer_id": { "type": "keyword" },
      "description": { "type": "text", "analyzer": "standard" },
      "total_amount": { "type": "double" },
      "created_at": { "type": "date" }
    }
  }
}

POST /_reindex?wait_for_completion=true&slices=auto
{
  "source": {
    "index": "orders-v1"
  },
  "dest": {
    "index": "orders-v2",
    "op_type": "create"
  }
}

Dual-write an application revision to both indices. Version conflicts caused by concurrent dual writes are expected in the backfill response; other failures abort cutover:

POST /orders-v1/_update/order-123
{
  "doc": {
    "description": "updated description",
    "total_amount": 125.50
  },
  "doc_as_upsert": true
}

POST /orders-v2/_update/order-123?version=42&version_type=external_gte
{
  "doc": {
    "description": "updated description",
    "total_amount": 125.50
  },
  "doc_as_upsert": true
}

Validate basic destination state; application or outbox metrics must separately prove that synchronization lag is zero:

GET /orders-v2/_count

GET /_cat/indices/orders-v1,orders-v2?v

GET /_cluster/health/orders-v2?wait_for_status=yellow&timeout=30s

Perform the atomic alias cutover:

POST /_aliases
{
  "actions": [
    {
      "remove": {
        "index": "orders-v1",
        "alias": "orders-read"
      }
    },
    {
      "remove": {
        "index": "orders-v1",
        "alias": "orders-write"
      }
    },
    {
      "add": {
        "index": "orders-v2",
        "alias": "orders-read"
      }
    },
    {
      "add": {
        "index": "orders-v2",
        "alias": "orders-write",
        "is_write_index": true
      }
    }
  ]
}

Production Safeguards and Differences

  • Lost or stale documents: caused by unresolved backfill races or unsynchronized deletes; prevent them with create semantics, durable events, and revision ordering.

  • Cluster pressure: monitor disk watermarks, shard allocation, rejected indexing tasks, JVM heap, reindex duration, and task failures. Throttle or pause the migration when necessary.

  • Recovery: snapshot the source, retain the old index through the rollback and retention periods, and test restoration beforehand.

  • Security: use TLS, least-privilege RBAC, restricted aliases, and protected administrative endpoints.

  • Lifecycle differences: Elasticsearch uses Index Lifecycle Management, while OpenSearch uses Index State Management. A typical Elasticsearch vector mapping is dense_vector; OpenSearch commonly uses knn_vector with its k-NN plugin.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 78 Elasticsearch answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Cloud, DevOps & Infrastructure subjects