Skip to content
Cloud, DevOps & Infrastructure

Apache Kafka interview questions and answers

80 questions, 25 with code. About 7 h 21 min to read them all.

By TestLab · Updated

Plan for an interview
80 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What is Apache Kafka and why is it used?

Apache Kafka is a distributed, durable, high-throughput event streaming platform that acts as a replicated commit log for real-time data pipelines and messaging between systems.

Read full answer

Applies toApache Kafka 3.3+ (KRaft production-ready; ZooKeeper removed in 4.0)

What It Is

Kafka stores streams of records in topics, which are split into partitions spread across brokers. Producers append records to partitions; consumers read them in order. Partitions are replicated for fault tolerance, and offsets track each consumer’s position, making reads replayable and decoupled from writes.

Problems It Solves

  • Decoupling: producers and consumers don’t need to be online or aware of each other.

  • Buffering and backpressure: Kafka absorbs traffic spikes so downstream systems can process at their own pace.

  • Durable messaging: data persists on disk with configurable retention, unlike in-memory queues.

  • Real-time pipelines: stream processing, log aggregation, CDC, and event-driven architectures.

  • Replayability: consumers can re-read historical data by resetting offsets.

Core Architecture

  • Broker: a Kafka server holding partitions.

  • Topic / Partition: logical stream split into ordered, immutable partitions.

  • Replication: each partition has a leader and in-sync replicas (ISR); failover promotes an ISR.

  • Consumer Group: consumers share partitions for parallel processing; each partition goes to one member.

  • KRaft: Kafka’s built-in Raft consensus for metadata management; ZooKeeper was removed in Kafka 4.0.

Guarantees and Trade-offs

Guarantee How Cost
Ordering Per partition only Must key related events to same partition
At-least-once Default producer/consumer Consumers must be idempotent
Exactly-once Idempotent producer + transactions Higher latency, transactional overhead
Durability acks=all + replication Higher write latency

Production Considerations

  • Monitoring: track consumer lag, ISR shrinkage, under-replicated partitions, and request latency via JMX/Prometheus.

  • Retention vs compaction: time/size retention for streams; log compaction for changelog/state topics.

  • Security: TLS in transit, SASL authentication, ACLs per topic; encrypt at rest via disk encryption.

  • Upgrades: rolling broker restarts with careful version compatibility; test client upgrades first.

  • Disaster recovery: MirrorMaker 2 for cross-cluster replication; back up topic configs and consumer offsets.

  • Common pitfalls: too few partitions (can’t scale consumers), hot partitions from poor keying, large messages, and ignoring consumer lag alerts.

When to Use

Use Kafka for high-volume, durable, replayable event streams and decoupled microservices. Prefer a simpler queue (RabbitMQ, SQS) for low-volume task queues or complex routing, and a database for transactional state.

2

How does Apache Kafka differ from RabbitMQ?

Apache Kafka is a partitioned, replicated event log, while RabbitMQ is a message broker built around exchanges, queues, acknowledgements, and per-message routing. Kafka favors durable retention, replay, and high-throughput streaming; RabbitMQ favors flexible routing and conventional work-queue delivery.

Read full answer

Applies toApache Kafka 4.x; RabbitMQ 4.x

Architecture and Data Model

Kafka stores records in topics, each divided into partitions. A partition is an append-only log with monotonically increasing offsets.

  • Brokers and replication: Brokers hold partition replicas. Each partition has one leader and normally one or more follower replicas.

  • Consumer groups: Members divide partitions between them. One group member consumes a partition at a time, and ownership changes through rebalancing.

  • KRaft: Kafka 4.x uses a KRaft controller quorum for metadata and partition-leader management rather than ZooKeeper.

  • Offsets: Consumers track their progress with committed offsets, allowing another consumer or group to replay the log independently.

RabbitMQ routes messages through exchanges into queues. Consumers receive queued messages and acknowledge them; unacknowledged messages can be redelivered. Exchanges support direct, topic, fanout, and headers-based routing.

Delivery, Ordering, and Offsets

Kafka delivery semantics depend on application and producer configuration:

  • At-most-once: Commit offsets before processing; a failure can lose unprocessed work.

  • At-least-once: Process first and commit afterward; a failure between those steps can cause duplicate processing.

  • Exactly-once within Kafka: For consume-transform-produce, use a transactional producer, include consumed offsets in the same transaction, and configure consumers with isolation.level=read_committed. This does not make arbitrary external side effects exactly-once.

  • Ordering: Kafka preserves order within a partition, not across a topic. Related records require the same key and an unchanged partition count to remain on the same partition.

RabbitMQ normally uses explicit acknowledgements, negative acknowledgements, prefetch limits, and redelivery. Publisher confirms indicate that a broker accepted a publish; they do not confirm consumer processing.

For reliable Kafka processing, disable automatic offset commits and commit only after successful handling. enable.idempotence=true with acks=all prevents retry duplicates and protects acknowledged writes from broker retries.

Feature Comparison

Area Apache Kafka RabbitMQ
Primary model Replicated partitioned log Exchanges, queues, and acknowledgements
Consumption Consumer groups and committed offsets Competing consumers and acknowledgements
Message lifecycle Retained until retention or compaction removes it Removed after acknowledgement; unacknowledged messages may be redelivered
Routing Topic plus key or partition selection; no native exchange-style routing Direct, topic, fanout, and headers exchanges
Ordering Per partition Queue insertion order per publishing channel; completion order can differ with concurrent consumers
Delivery At-most-once, at-least-once, or transactional exactly-once within Kafka At-most-once or at-least-once consumption
Primary scaling Partitions and independent consumer groups Queue distribution and consumer concurrency

Retention, Compaction, and Replay

Kafka applies retention policies independently to topics:

  • delete: Removes old segments according to age or size limits.

  • compact: Eventually retains the latest record for each key and removes older records with that key.

  • compact,delete: Combines compaction with normal retention.

This replayable history supports event sourcing, state rebuilding, and reprocessing. RabbitMQ queues are primarily delivery structures rather than long-term archives. Durable queues survive broker restart, while quorum queues replicate messages across RabbitMQ nodes for higher availability.

Operations, Failures, and Security

Monitor Kafka consumer lag, under-replicated and offline partitions, broker disk utilization, request latency, controller health, and JVM or garbage-collection pressure. Consumer lag is generally more useful than queue depth because Kafka deliberately retains consumed records.

  • Broker loss: The KRaft controller elects an in-sync replica as leader. Maintain adequate replication-factor and min.insync.replicas; without an in-sync replica, an unclean election may lose data.

  • Disk exhaustion: Retention can fall behind, so alerts should forecast disk capacity rather than rely only on current usage.

  • Rebalances: Long processing or frequent membership changes interrupt progress. Bound processing and use cooperative assignment where appropriate.

  • Hot partitions: A skewed key can overload one partition.

  • Poison records: Use bounded retries and route repeatedly failing records to a dead-letter topic.

Kafka replication provides availability, not backup. Use a tested snapshot/export or object-store backup process for logical recovery; MirrorMaker 2 provides cross-cluster replication, but replicated deletions or corruption can also propagate. Rolling upgrades must follow Kafka’s supported compatibility sequence.

Secure Kafka with TLS, SASL such as SCRAM-SHA-512, and topic ACLs. Monitor RabbitMQ queue depth, unacknowledged deliveries, memory or disk alarms, node health, and quorum-queue failover; RabbitMQ also provides TLS, authentication, and authorization controls.

Ecosystem and Choosing One

Apache Kafka provides Kafka Connect for source and sink connectors and Kafka Streams for stateful stream-processing applications. Connect exactly-once support remains connector-specific. Schema Registry is separate from Apache Kafka; Confluent Schema Registry or a compatible implementation manages Avro, JSON Schema, and Protobuf schemas.

Choose Kafka when events must be retained, replayed, partitioned across many consumers, or integrated with stream processing. Choose RabbitMQ when flexible routing, acknowledgements, priorities, dead-letter exchanges, request/reply patterns, and conventional task queues matter more than long-term log retention.

3

How do Kafka brokers, topics, and partitions relate?

In Apache Kafka, a broker stores and serves Kafka data, a topic is a logical stream of records, and a partition is the ordered, append-only log within a topic. Together, they determine how data is distributed, replicated, consumed, and scaled.

Read full answer

Applies toApache Kafka 3.9+ (KRaft mode)

Core Components

Component Responsibility Production implications
Broker A Kafka server that stores partition data and serves producer and consumer requests Multiple brokers provide capacity and failure isolation; each broker is a separate failure domain
Topic A named category or stream of records, such as orders or payment-events Topics are logical; their data is physically stored in partitions
Partition An ordered sequence of records identified by monotonically increasing offsets Partitions are the unit of parallelism, replication, retention, and consumer assignment
Leader The broker replica currently accepting writes and serving that partition Producers send records to the leader, which replicates them to followers
Followers Broker replicas that maintain copies of the partition If the leader fails, an in-sync follower can become the new leader

A topic such as orders might contain 12 partitions spread across three brokers. Each partition is replicated three times, so the topic has a replication factor of three and 36 physical replica directories across the cluster, excluding any additional recovery or archival copies.

Partitions, Ordering, and Consumer Groups

Kafka guarantees record ordering only within a partition. It does not provide a global ordering across partitions.

  • Partition selection: A producer sends keyed records to the same partition when the partition count and key remain stable. The producer hashes the key using the configured partitioner.

  • Parallel processing: Different consumer-group members process different partitions of the same topic, allowing processing to scale with partition count.

  • Consumer offsets: Each consumer group tracks the next position to read for every assigned partition. These offsets are stored in Kafka’s internal __consumer_offsets topic.

  • Rebalancing: When members join, leave, or exceed the configured max.poll.interval.ms, Kafka reassigns partitions. Frequent rebalances can cause consumer lag or duplicate processing.

  • Partition count: Increasing it can increase throughput but changes the partitioning of keyed records. Kafka does not support decreasing a topic’s partition count.

To preserve ordering, producers should send related records to the same partition rather than trying to coordinate ordering across Kafka.

Replication and Delivery Guarantees

A typical production configuration uses at least three replicas:

  • replication.factor=3: Store each partition on three brokers.

  • min.insync.replicas=2: Require two in-sync replicas for a successful write.

  • Producer acks=all: Wait for the leader and required followers to acknowledge the record.

With three replicas and min.insync.replicas=2, the topic remains writable if one broker fails. If two replicas leave the in-sync replica set, writes should fail instead of silently reducing durability.

Kafka’s at-least-once processing is achieved by committing a consumer offset only after processing succeeds. A crash between processing and committing causes the record to be replayed. Exactly-once behavior requires additional mechanisms: an idempotent producer, Kafka transactions where applicable, isolation.level=read_committed on consumers, and exactly-once-capable source connectors. Exactly-once Kafka transactions do not automatically make arbitrary external side effects exactly once.

Retention and Compaction

Kafka manages storage primarily through log segments rather than deleting individual records immediately.

  • Time retention: retention.ms or retention.bytes causes old segments to be deleted.

  • Size retention: The oldest segments are removed when the configured byte limit is exceeded.

  • Compaction: With cleanup.policy=compact, Kafka retains the latest record for each key rather than every historical value.

  • Tombstones: A record with a null value marks a key as deleted, but the tombstone itself remains until retention or compaction removes it.

Use compaction for state-like streams where only the latest value matters. Use normal retention for immutable event logs. Configure retention according to replay, recovery, and regulatory requirements rather than simply filling available disk.

Kafka Ecosystem

  • Kafka Connect runs source and sink connectors for moving data between Kafka and systems such as databases, object stores, and search engines. Connector concurrency scales independently from broker capacity.

  • Kafka Streams is a Java library for processing streams inside an application. It supports stateful operations, local state stores, and changelog topics.

  • Schema Registry stores and validates schemas, commonly Avro, Protobuf, or JSON Schema. Schema compatibility rules prevent producers from introducing incompatible record shapes.

  • KRaft replaces ZooKeeper for cluster metadata. In modern Kafka clusters, dedicated controller nodes form the metadata quorum, while brokers store data and serve clients.

Operations, Security, and Recovery

Inspect topology with kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders. This shows the leader, replicas, in-sync replicas, and partition assignments. Consumer health can be checked with kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processors.

Monitor under-replicated and offline partitions, ISR changes, producer errors, request latency, disk usage, JVM pressure, controller health, and consumer lag. Alert on sustained ISR shrinkage rather than treating every transient broker restart as a data-loss incident.

Replication is not a backup: it can replicate accidental deletions or corruption. Use cross-cluster replication, an appropriate Kafka Connect sink, or object-storage archival for disaster recovery. Protect Kafka with TLS and SASL, least-privilege ACLs, encrypted storage where required, and separate listener configurations for clients, brokers, and controllers. Perform broker upgrades as rolling operations, verify client compatibility, and confirm replication health before continuing each stage.

4

What is an Apache Kafka cluster?

An Apache Kafka cluster is a distributed system of Kafka brokers that stores ordered, replicated event streams in topics. Producers publish records to partitions, while consumers read them independently or as members of consumer groups.

Read full answer

Applies toApache Kafka 3.9 and 4.x with KRaft mode

Core Architecture

A production Kafka cluster normally separates two responsibilities:

  • Brokers: Store partition data, serve producer and consumer requests, replicate records, and perform leader elections.

  • KRaft controllers: Maintain cluster metadata such as topics, partitions, replication assignments, and broker membership. A quorum of controllers prevents conflicting metadata changes.

A topic is divided into partitions, which are the unit of parallelism and storage. Each partition has one leader broker and zero or more follower brokers.

Component Purpose
Producer Sends records to a topic, optionally using keys for partition selection
Broker Stores partitions and serves client requests
Partition Append-only, ordered log segment
Consumer Reads records from assigned partitions
Consumer group Distributes partitions among consumers and coordinates offset commits
KRaft controller quorum Manages and distributes cluster metadata

Clients initially contact a broker through the bootstrap.servers setting. That broker returns metadata containing the leaders for the required partitions, after which clients connect directly to the appropriate leaders.

Replication and Durability

Each partition is replicated across multiple brokers. Acknowledgement settings and min.insync.replicas determine whether a write is considered safe.

For strong durability:

  • Use at least three brokers for important topics.

  • Set an appropriate replication factor, commonly 3.

  • Configure min.insync.replicas to 2.

  • Use acks=all for critical producer writes.

  • Ensure the producer can handle retriable errors and configure delivery timeouts appropriately.

With acks=all, if the in-sync replica count falls below min.insync.replicas, produce requests fail because too few replicas are in sync. Using acks=1 does not provide this guarantee because the leader can acknowledge a write after appending it locally without waiting for a follower.

When a leader fails, a follower in the in-sync replica set can become the new leader. Replication protects against broker or disk failure, but not against deleting a topic or corrupting data on every replica. Production clusters therefore also need backups, appropriate ACLs, and tested recovery procedures.

Ordering and Delivery Semantics

Kafka guarantees record ordering within a partition, not across an entire topic. Records with the same key usually go to the same partition, preserving their relative order.

  • At-most-once: The producer does not retry failed sends, and consumers commit offsets before processing. Records may be lost but should not normally be delivered twice.

  • At-least-once: The producer retries and consumers commit offsets only after successful processing. Duplicate processing is possible, so consumers should be idempotent.

  • Exactly-once processing within Kafka: Enable idempotent production (enable.idempotence=true), use a unique transactional.id, write output records and consumed offsets in the same producer transaction, and consume with isolation.level=read_committed. acks=all is also required for idempotence. This supports atomic consume-transform-produce flows but does not make arbitrary external side effects exactly once.

Consumer groups assign at most one active consumer to each partition within a group. Adding consumers helps only while partitions remain available.

Storage and Data Lifecycle

Kafka stores records in immutable segment files. Retention policies control disk usage:

  • Time-based retention (retention.ms): Records become eligible for deletion after the configured age. Kafka deletes eligible rolled segments; it does not immediately remove records from an active segment.

  • Size-based retention keeps a bounded amount of data per partition.

  • Log compaction retains the latest record for each key, useful for state-like topics.

  • Delete retention controls how long compaction tombstones remain available to remove older records.

Partition count determines maximum consumer-group parallelism and influences broker load. Increasing it adds partitions and can change the default key-to-partition mapping, so partition growth should be planned carefully. Records already written remain in their original partitions.

Supporting Components and Operations

  • Kafka Connect runs source and sink connectors for moving data between Kafka and systems such as databases, search indexes, and cloud services.

  • Kafka Streams builds stateful or stateless stream-processing applications directly in Java or other supported runtimes.

  • Schema Registry manages schemas and compatibility, commonly with Avro, Protobuf, or JSON Schema.

Production monitoring should cover under-replicated partitions, offline partitions, leaders without followers, request latency, consumer lag, disk usage, controller quorum health, JVM pressure, and broker connectivity. Security should use TLS encryption and SASL authentication with topic-level ACLs rather than relying on network isolation alone.

Kafka upgrades should follow the supported rolling-upgrade procedure. Before upgrading, verify broker/controller compatibility, listener configuration, JVM requirements, and the target release’s migration requirements. In KRaft mode, format the metadata quorum only during initial cluster creation or the prescribed metadata migration—not on every restart.

5

How does KRaft replace ZooKeeper in Apache Kafka?

KRaft (Kafka Raft) is Kafka’s built-in consensus protocol that stores cluster metadata in an internal replicated log instead of an external ZooKeeper ensemble, removing ZooKeeper as a dependency entirely.

Read full answer

Applies toApache Kafka 3.3+ (KRaft production-ready for new clusters); ZooKeeper removed in Kafka 4.0.

What KRaft Is

KRaft implements the Raft consensus algorithm inside Kafka itself. Cluster metadata (topics, partitions, configurations, ACLs, broker registrations) is written to a special internal topic, __cluster_metadata, which is replicated across a set of controller nodes forming a quorum. One controller is elected active leader; the rest are hot standbys. This makes metadata management a first-class part of Kafka rather than an external service.

How It Replaces ZooKeeper

ZooKeeper previously held the source of truth for broker membership, topic metadata, controller election, and configuration. KRaft moves all of that into Kafka’s own replicated log:

  • Controller election happens via Raft leader election among controller quorum voters, not via ZooKeeper ephemeral nodes.

  • Metadata propagation uses the replicated metadata log; brokers fetch metadata updates from the active controller like they would fetch topic data.

  • Broker registration is recorded in metadata records rather than ZooKeeper znodes.

  • Single security model: one TLS/SASL configuration for the whole cluster instead of separate ZooKeeper security.

Architecture and Components

  • Controller quorum: a small set of nodes (typically 3 or 5) running in controller or combined mode, voting on metadata changes.

  • Metadata log: __cluster_metadata, an internal topic with its own partitions and replication, snapshotted periodically for fast recovery.

  • Broker nodes: run in broker mode and receive metadata from controllers; in small deployments a node can run in combined mode as both broker and controller.

  • KRaft mode flag: process.roles=broker,controller (combined) or split roles for production.

Operational Differences

Aspect ZooKeeper mode KRaft mode
External dependency ZooKeeper ensemble required None
Controller failover Seconds to tens of seconds Sub-second to few seconds
Metadata scalability ZK znode limits Log-based, much larger scale
Security config Separate ZK auth Unified Kafka auth
Upgrade path In-place migration supported Target state for Kafka 4.0

Migration and Compatibility

Existing ZooKeeper-based clusters can migrate to KRaft using the metadata migration tooling, including kafka-metadata-quorum.sh for quorum status, with the migration process documented for Kafka 3.5+. Kafka 4.0 removed ZooKeeper support entirely, so new clusters should be deployed in KRaft mode from the start. Snapshots in the metadata log make controller restarts and recovery significantly faster than ZooKeeper-based recovery.

Best Practices and Pitfalls

  • Use an odd number (3 or 5) of controller voters for quorum; never run a single controller in production.

  • Separate controller and broker roles on distinct nodes for large clusters to isolate metadata traffic.

  • Keep controller.quorum.voters identical across all nodes; mismatches prevent quorum formation.

  • Monitor kafka.controller metrics and metadata log lag; a lagging follower controller delays failover.

  • Do not mix ZooKeeper and KRaft metadata in the same cluster; migration is one-directional and must complete fully.

  • Back up by replicating the metadata log via normal Kafka replication; there is no separate ZooKeeper backup to manage, but snapshot directories under log.dirs should be included in node backups.

6

What is an offset in Apache Kafka?

An offset in Kafka is a monotonically increasing integer assigned to each record within a partition, serving as its position in that partition’s append-only log. Consumers track their progress by committing offsets, allowing a consumer group to resume from its last committed position after restarts, rebalances, or failures.

Read full answer

Applies toApache Kafka 4.x (KRaft mode)

How It Works

  • Each partition is an independent append-only log. Records normally begin at offset 0, and clients do not choose offsets; the broker assigns them as records are appended.

  • Offsets are per partition, not per topic. A record is uniquely identified by the tuple (topic, partition, offset).

  • The log end offset (LEO) is the next offset that will be assigned. The high watermark (HW) marks the latest record fully replicated to all current in-sync replicas.

  • The default isolation level, read_uncommitted, can expose records through the LEO. A read_committed consumer instead stops at the last stable offset (LSO), excluding records from transactions that have not completed.

  • Offsets are immutable positions. Retention deletion and log compaction can leave gaps, but surviving records are not renumbered.

  • KRaft replaces ZooKeeper-based controller metadata management; it does not change offset semantics. Consumer-group offsets are still stored in Kafka’s internal __consumer_offsets topic.

Consumer Offsets and Committing

  • The __consumer_offsets topic stores the committed offset for each consumer group and topic-partition.

  • A committed offset represents the next record to consume. Committing offset 42 means that consumption should resume at 42, after processing through 41.

  • Auto-commit is controlled by enable.auto.commit, whose default is true, and commits positions returned by consumer polls in the background. Java consumers can use commitSync() or commitAsync() for manual control.

  • The broker setting offsets.retention.minutes controls how long offsets are retained; its default is 10080 minutes, or seven days. If a group does not commit an offset during that period, the offset can expire.

  • When a committed offset is missing or out of range, the consumer initializes according to auto.offset.reset: latest by default, or earliest when configured.

Delivery Semantics and Ordering

  • Kafka preserves record order within a partition, and consumers observe those records in offset order. It provides no ordering guarantee across partitions.

  • At-least-once processing is obtained by committing only after successful processing. A crash after processing but before committing can cause the same record to be processed again.

  • Exactly-once processing within Kafka requires a transactional producer with a stable transactional.id, transactional publication of output records and consumer offsets using sendOffsetsToTransaction(), and consumers configured with isolation.level=read_committed. Idempotence alone is not sufficient, and transactions do not atomically cover external database or HTTP side effects.

Operational Considerations

  • Consumer lag is a derived metric rather than a value stored by the broker. It is commonly calculated from the group’s committed or current position and the partition’s log-end position. During active processing, committed lag can overstate the consumer’s in-process lag; for read_committed consumers, the LEO can also overstate useful application-record lag because it includes control records and records from incomplete transactions.

  • Inspect committed offsets and lag with a runnable command such as bin/kafka-consumer-groups.sh --bootstrap-server broker.example.com:9092 --describe --group my-group.

  • Reset offsets only while the group has no active members. For example: bin/kafka-consumer-groups.sh --bootstrap-server broker.example.com:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute. The tool supports --to-earliest, --to-latest, --to-datetime, and --shift-by.

  • A committed offset below the partition’s log start offset is out of range after retention deletion. The consumer handles it according to auto.offset.reset; if the policy is none, consumption cannot proceed until an operator supplies a valid position.

  • On compacted topics, offsets are not contiguous and therefore cannot be interpreted as record counts.

Best Practices

  • Use manual commits after successful processing to define replay boundaries, while accepting that duplicates remain possible between processing and commit.

  • Make consumers idempotent, or use a stable event identifier for deduplication, when downstream side effects may be repeated.

  • Alert on per-partition lag and the age of the oldest unprocessed record rather than relying only on aggregate lag.

  • Keep offsets.retention.minutes above the longest expected period of consumer inactivity when uninterrupted group resumption is required.

  • Use Kafka transactions and read_committed only when end-to-end exactly-once behavior is required, and design separately for external side effects that Kafka transactions cannot cover.

7

How does Apache Kafka store messages on disk?

Kafka stores messages as append-only partition logs organized into immutable log segments. Each topic is split into partitions, and every partition replica maintains its own on-disk log containing message batches, offset indexes, and timestamp indexes.

Read full answer

Applies toApache Kafka 4.x in KRaft mode

Disk Layout and Record Format

A broker stores partition data beneath the directories configured by log.dirs. Each partition replica normally has a directory named using the topic name, partition number, and a unique replica identifier.

A partition contains files such as:

  • Log segment (.log): Stores compressed message batches in append-only order.

  • Offset index (.index): Maps logical offsets to physical positions in log segments, allowing efficient lookup without scanning the entire segment.

  • Timestamp index (.timeindex): Provides a sparse timestamp-to-offset mapping for timestamp-based offset lookups.

  • Leader epoch checkpoint (leader-epoch-checkpoint): Records changes in partition leadership, helping followers validate leader history during replication.

Messages are written as record batches. A batch contains multiple records and metadata such as the base offset, timestamps, producer ID and epoch, base sequence, leader epoch, compression settings, and a CRC32C checksum. Compression can be applied to the records within a batch, reducing disk usage and network transfer at the cost of additional CPU usage.

Kafka uses the operating system’s page cache to buffer writes. It does not normally perform an fsync for every producer request; replication protects against the loss of an individual replica, while the operating system controls when dirty pages reach persistent storage.

Partitioning and Replication

Messages within a partition retain their append order, but Kafka provides no ordering guarantee across different partitions. A producer chooses the destination partition, usually by key hashing; the same key therefore maps consistently to one partition unless the partition count changes or a custom partitioner behaves differently.

Every partition has one leader and zero or more follower replicas. Only the leader accepts writes, and followers replicate those batches. With a replication factor of three, a complete copy of each partition is generally maintained on three brokers.

KRaft stores Kafka’s cluster metadata, including topic and partition leadership information, in a separate internal metadata log. It does not replace the per-partition message logs.

Retention and Log Compaction

Kafka controls disk usage with two primary topic-level mechanisms:

  • Delete retention: cleanup.policy=delete removes whole log segments after the configured age or size limits expire. Active segments are normally rolled before they can be removed, so actual retention can exceed the configured value.

  • Log compaction: cleanup.policy=compact removes older records when a newer record with the same key exists in the same partition. The latest value for a key is not necessarily removed immediately, and compaction does not guarantee that exactly one physical record remains for every key.

Important retention configurations include:

  • retention.ms: Maximum age of records.

  • retention.bytes: Maximum retained data per partition.

  • segment.bytes: Maximum size of a log segment before rolling.

  • segment.ms: Maximum age of a log segment before rolling.

  • cleanup.policy: Controls deletion, compaction, or both.

A record with a null value is a tombstone. During compaction, it removes the previous non-null value for that key, but the tombstone itself remains until delete-retention processing removes it.

Consumer group offsets are stored separately in the internal __consumer_offsets topic, which is compacted and replicated. Business-message storage and consumer offset storage should not be treated as the same mechanism.

Durability and Failure Modes

For production durability, use a replication factor of at least three, configure producers with acks=all, and set min.insync.replicas appropriately. For example, with three replicas, min.insync.replicas=2 rejects writes when fewer than two replicas are available.

Important failure scenarios include:

  • Broker loss: A follower can be recreated from the leader. If the leader fails, another in-sync replica can become leader.

  • Unclean leader election: Keep unclean.leader.election.enable=false to prevent an out-of-sync replica becoming leader and silently losing records.

  • Disk exhaustion: Retention failures, oversized segments, compaction backlog, or insufficient disk can stop writes. Capacity should account for replication: three replicas require approximately three times the retained message data.

  • Corrupted segment: Kafka validates record checksums, but operators still need tested procedures for restoring or rebuilding affected replicas.

  • Long garbage-collection or I/O pauses: These can increase producer latency, request timeouts, and consumer lag.

Operational Practices

Inspect partition health with commands such as bin/kafka-topics.sh --bootstrap-server 10.0.0.10:9092 --describe --topic orders --under-replicated-partitions. Review topic retention with bin/kafka-configs.sh --bootstrap-server 10.0.0.10:9092 --describe --entity-type topics --entity-name orders.

Production monitoring should include:

  • Disk usage, free space, and I/O latency per broker.

  • Under-replicated and offline partition counts.

  • Broker log flush errors and failed requests.

  • Producer retry and delivery-failure rates.

  • Consumer lag and offsets for critical consumer groups.

  • Retention deletion and compaction backlogs.

Plan capacity using peak ingestion, replication factor, retention, compression, and segment overhead. Before upgrades or broker replacements, verify that replicas are healthy and that the replacement broker has sufficient disk and network capacity.

8

What are log segments in Apache Kafka?

A log segment is a file-backed segment of an Apache Kafka partition’s append-only log. Each partition consists of one or more segments, allowing Kafka to roll active data into separate files and reclaim space without rewriting the whole partition.

Read full answer

Applies toApache Kafka 3.6–4.x

How Segments Work

  • A topic is divided into partitions, and each partition has an independent ordered log.

  • The active segment is the file to which the broker currently appends record batches.

  • Kafka rolls the active segment when a configured size or time threshold is reached and starts a new segment.

  • A segment filename begins with the segment’s base offset, the offset of its first record. Kafka normally zero-pads this value to 20 digits, for example 00000000000000000123.log.

  • Companion .index and .timeindex files map offsets and timestamps to physical positions, allowing Kafka to locate records without scanning an entire segment.

  • The index is generally sparse, so it does not contain an entry for every record.

  • Segments are local files on each broker. Leaders append records, while follower replicas maintain corresponding local log data.

Segments are an internal storage detail. Producers write to partitions, and consumers fetch records by offset; neither normally interacts with segment files directly. Crossing a segment boundary does not create a transaction boundary or change delivery semantics.

Retention and Compaction

Kafka normally applies retention at segment granularity rather than deleting individual records.

  • cleanup.policy=delete: Removes old segments when the configured time or size threshold is reached. If both are configured, either threshold can trigger deletion.

  • log.retention.ms: Maximum time that log data is retained when time-based retention applies.

  • log.retention.bytes: Maximum retained size per partition when size-based retention applies.

  • log.segment.bytes: Segment-size threshold used when rolling the active segment.

  • log.roll.ms: Time-based threshold for rolling the active segment.

  • log.roll.jitter.ms: Randomizes time-based segment rolls, reducing synchronized roll activity across partitions.

With cleanup.policy=compact, the log cleaner removes obsolete records for keys that it processes. The compacted state exposes the latest value for each key, although Kafka does not guarantee that every physical segment contains at most one record for that key.

A null-valued record is a tombstone. It causes older values for that key to be removed during compaction and marks the key as deleted for compacted-state consumers. The tombstone itself may be physically removed only after the configured delete.retention.ms interval.

Compaction operates within a partition. Records for a key must therefore use the same partition for compaction to provide the intended result.

Delivery Guarantees

Kafka’s ordering guarantee applies within a partition, including across segment boundaries. There is no ordering relationship between different partitions.

Mechanism Guarantee and requirements
At-least-once consumption Process the record before committing its offset. Disable automatic commits or otherwise commit only after successful processing. A crash after processing but before committing can cause reprocessing.
Idempotent production enable.idempotence=true prevents retry duplicates within the producer’s idempotence session. It does not provide end-to-end exactly-once processing.
Transactional exactly-once For Kafka consume-transform-produce flows, use a transactional producer with a unique transactional.id, send records and consumer offsets in the same transaction, and call commitTransaction(). Consumers must use isolation.level=read_committed to avoid observing uncommitted or aborted transactional records.

A segment boundary does not affect any of these guarantees.

Operational Considerations

  • Disk pressure: Partition count, replication, retention, and delayed segment cleanup can exhaust broker disks. log.segment.bytes controls segment rolling, not total partition storage.

  • Consumer lag: Retention may remove records before a lagging consumer reads them. If its committed offset is below the log-start offset, offset-reset behavior can move it forward and silently skip expired data.

  • Replication: Kafka replicates records and batches to followers over its network protocol; it does not replicate raw directory files. Monitor under-replicated and offline partitions.

  • KRaft: Controllers maintain cluster metadata, while partition data remains on brokers. Controller quorum health and broker disk capacity are therefore separate operational concerns.

  • Security: Segment files contain record data and may expose sensitive information. Restrict filesystem access, use disk encryption where required, and protect broker communication with TLS and mechanisms such as SASL.

Production Best Practices and Failure Modes

  • Size partitions using expected throughput, retention period, consumer parallelism, and replication factor.

  • Maintain sufficient disk headroom for segment rolls, compaction, replicas, and temporary recovery activity.

  • Alert on disk utilization, under-replicated partitions, offline partitions, request latency, broker errors, and consumer lag.

  • Treat a sudden lag reduction as potentially suspicious; it may indicate expired data and an offset reset rather than healthy catch-up.

  • Do not manually delete, rename, or edit segment files while Kafka is running. Let Kafka perform retention and compaction, or stop the affected broker and follow Kafka’s documented recovery procedures.

9

How does replication work in Apache Kafka?

Replication in Apache Kafka is the mechanism by which each partition of a topic is copied across multiple brokers so that data survives broker failures and reads/writes can continue without data loss.

Read full answer

Applies toApache Kafka 3.7+ (KRaft mode)

How It Works

Each topic is split into partitions, and every partition is replicated across a configurable number of brokers. One replica is designated the leader; by default, all produce requests and consumer fetches for that partition go to the leader. The other replicas are followers that pull data from the leader and stay in sync. If the leader fails, one of the in-sync followers is elected as the new leader.

Key Components

  • Replication factor: number of copies of each partition (commonly 3 in production).

  • Leader: the replica that serves client traffic for a partition by default.

  • Followers: replicas that fetch from the leader; with rack-aware follower fetching enabled, they can also serve some consumer reads.

  • ISR (In-Sync Replicas): the set of replicas (leader + followers) that are caught up within replica.lag.time.max.ms.

  • min.insync.replicas: minimum number of replicas that must acknowledge a write when acks=all.

Durability and Acknowledgements

Producers choose durability via acks:

acks value Behaviour Durability
0 No acknowledgement; fire and forget Lowest; data can be lost
1 Leader acknowledges after writing to its log Medium; leader failure can lose unreplicated data
all (-1) Leader waits for all ISR replicas to acknowledge Highest; survives leader failure

With acks=all and min.insync.replicas=2 on a replication factor of 3, a write succeeds only if at least 2 in-sync replicas, including the leader, acknowledge it. If the ISR shrinks below min.insync.replicas, writes fail with NOT_ENOUGH_REPLICAS rather than silently weakening durability.

Failure Handling

  • If a follower falls behind beyond replica.lag.time.max.ms, it is removed from the ISR; acks=all then waits for the smaller ISR, subject to min.insync.replicas.

  • If the leader fails, the KRaft controller elects a new leader from the ISR. Replicas outside the ISR are not eligible, which prevents data loss but can make a partition temporarily unavailable if no in-sync follower remains.

  • Unclean leader election (unclean.leader.election.enable=true) allows an out-of-sync replica to become leader, trading availability for potential data loss; keep it disabled in production.

Ordering Guarantees

Ordering is guaranteed only within a single partition. Replication preserves order on the leader’s log, and followers apply records in the same order. Cross-partition ordering is not provided.

Operational Considerations

  • Monitoring: track UnderReplicatedPartitions, OfflinePartitionsCount, IsrShrinksPerSec/IsrExpandsPerSec, and ActiveControllerCount (should be 1).

  • Capacity: replication multiplies storage and network traffic by the replication factor; size brokers accordingly.

  • Rebalancing: adding brokers requires partition reassignment; use kafka-reassign-partitions.sh and throttle with --throttle to avoid saturating the cluster.

  • Backups: replication is not a backup; use MirrorMaker 2 for disaster recovery and tiered storage for long-term retention.

Best Practices

  • Use replication factor 3 with min.insync.replicas=2 for production topics.

  • Set acks=all for producers that require durability.

  • Keep unclean.leader.election.enable=false.

  • Monitor ISR shrinkage; sustained shrinkage indicates slow followers or network issues.

  • Place replicas across failure domains (racks/availability zones) using broker.rack and rack-aware assignment.

10

What are leader and follower replicas in Kafka?

In Kafka, each partition is replicated across multiple brokers for fault tolerance. Among the replicas of a partition, exactly one acts as the leader and the rest are followers. The leader is the single replica that serves all produce and fetch requests for that partition, while followers continuously pull data from the leader to stay in sync.

Read full answer

Applies toApache Kafka 3.x and 4.x (KRaft mode)

How Replication Works

When a topic is created with replication factor N, each partition gets N replicas placed on different brokers. One replica is elected leader; the others become followers. All client traffic for that partition goes to the leader, which keeps an ordered log. Followers replicate by issuing fetch requests to the leader, appending the same records in the same order.

  • Leader replica: handles every produce request and every consumer fetch for its partition; advances the high-watermark as followers catch up.

  • Follower replica: does not serve client requests by default; pulls records from the leader and writes them to its local log; participates in leader election when the leader fails.

In-Sync Replicas (ISR)

Followers that are caught up within replica.lag.time.max.ms (default 30s) are part of the ISR set. Only ISR members are eligible to become leader, which is what makes failover safe.

  • min.insync.replicas: minimum ISR size required for a produce with acks=all to succeed.

  • If ISR shrinks below this threshold, producers with acks=all receive NOT_ENOUGH_REPLICAS errors.

Leader Election and Failover

When a leader broker fails or is shut down, the controller (the KRaft quorum leader in 3.x/4.x) elects a new leader from the ISR. The new leader’s log becomes authoritative; any records not replicated to it are lost.

  • unclean.leader.election.enable=false (default): only ISR members can be elected; safer, but may cause unavailability if ISR is empty.

  • unclean.leader.election.enable=true: allows an out-of-sync replica to become leader; risks data loss but keeps the partition available.

Follower Reads

Since Kafka 2.4, consumers can fetch from the closest replica, including followers, when client.rack is set and replica.selector.class is configured on brokers. This reduces cross-AZ traffic but does not change which replica is authoritative for writes.

Key Configuration

Setting Scope Purpose
Replication factor Set at topic creation Number of replicas per partition
min.insync.replicas Topic or broker Minimum ISR for acks=all writes
replica.lag.time.max.ms Broker How long a follower can lag before leaving ISR
unclean.leader.election.enable Broker Allow out-of-sync leader election
default.replication.factor Broker Default replication factor for auto-created topics

Production Considerations

  • Spread replicas across racks or availability zones using broker.rack so a single failure domain does not take out all replicas of a partition.

  • Monitor UnderReplicatedPartitions, OfflinePartitionsCount, and IsrShrinksPerSec/IsrExpandsPerSec; sustained ISR shrinkage indicates a slow or failing follower.

  • Reassign partitions with kafka-reassign-partitions.sh after adding brokers; otherwise new brokers hold no replicas and old brokers stay overloaded.

  • Prefer acks=all with min.insync.replicas=2 on a 3-replica topic for durability; this tolerates one broker loss without data loss.

  • During rolling upgrades, followers catch up after restart; watch replication lag before moving to the next broker.

11

What are in-sync replicas (ISR) in Apache Kafka?

In-sync replicas (ISR) are the broker replicas of a Kafka partition that are active and sufficiently caught up with the partition leader to be trusted as synchronous copies. The leader normally belongs to the ISR, and ISR members are the preferred candidates for clean leader election.

Read full answer

Applies toApache Kafka 3.x and 4.x

ISR Membership

Each Kafka partition has one leader and one or more follower replicas:

  • The leader receives records from producers.

  • Followers fetch records from the leader and maintain local copies.

  • The controller tracks which replicas are sufficiently synchronized.

  • A follower that becomes unavailable or falls behind may be removed from the ISR.

  • Once the follower catches up, the controller can add it back to the ISR.

A replica can be operational but outside the ISR if it has fallen too far behind. The broker setting replica.lag.time.max.ms determines how long a replica may lag before being considered out of sync.

Replication and Acknowledgements

With acks=all, the leader waits for all replicas currently in the ISR to acknowledge the record before completing the producer request. It does not necessarily wait for every configured replica if some are outside the ISR.

The topic setting min.insync.replicas establishes the minimum acceptable ISR size:

Configuration Typical behavior Trade-off
replication.factor=3, min.insync.replicas=2, acks=all When all three replicas are in sync, writes wait for all three ISR members; if one replica drops out, writes can continue with two Tolerates one replica failure without rejecting writes, while still requiring two current copies
replication.factor=3, min.insync.replicas=3, acks=all Writes wait for all three ISR members and fail if any replica is out of sync Strongest protection against losing acknowledged records, but no replica-failure write tolerance
replication.factor=3, min.insync.replicas=1, acks=all Writes wait for all current ISR members, which may be only the leader Greatest availability, but weakest protection if the leader’s only current copy is lost

If the ISR falls below min.insync.replicas, producers using acks=all receive errors such as NOT_ENOUGH_REPLICAS. This prevents the cluster from silently acknowledging records that exist on too few replicas.

Guarantees and Consistency

ISR supports Kafka’s replication guarantees, but it does not by itself provide exactly-once processing:

  • Ordering: Kafka preserves record order within a partition, not across an entire topic.

  • At-least-once delivery: Consumers can commit offsets before or after processing and may process a record again after a crash.

  • Exactly-once processing: Requires idempotent or transactional producers, Kafka transactions, and consumers using isolation.level=read_committed.

  • Durability: acks=all combined with an appropriate min.insync.replicas value protects against losing acknowledged records during a broker failure.

For transactional workloads, the internal transaction-state topic should also use suitable replication and ISR settings.

Failure Handling

When a leader fails, a clean election normally selects a leader from the ISR. With the broker setting unclean.leader.election.enable=false, Kafka avoids promoting a stale replica that may have lost acknowledged records.

If no ISR member is available, the choices are:

  • Prioritize consistency: Keep the partition unavailable until a synchronized replica returns.

  • Prioritize availability: Enable unclean election, allowing a potentially stale replica to become leader.

The second option can cause data loss and should be used only when the availability tradeoff is explicit.

Production Operations

Inspect the leader, configured replicas, and current ISR with:

bin/kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders

The output includes Leader, Replicas, and Isr. Configure the minimum ISR size for a topic with:

bin/kafka-configs.sh --bootstrap-server kafka-1.example.com:9092 --entity-type topics --entity-name orders --alter --add-config min.insync.replicas=2

Monitor these broker metrics:

  • UnderReplicatedPartitions: Partitions with a configured replica missing from the ISR.

  • UnderMinIsrPartitionCount: Partitions whose ISR is smaller than min.insync.replicas.

  • OfflinePartitionsCount: Partitions without an available leader.

  • ActiveControllerCount: Should normally be 1 in a healthy cluster.

During rolling upgrades, brokers restart one at a time, which can temporarily shrink the ISR. Stagger upgrades and avoid an overly aggressive min.insync.replicas, or producers may be blocked. Place replicas across separate brokers and failure domains, and do not treat ISR as a backup: ISR protects against failures within the Kafka cluster, but it does not replace off-cluster replication or independent backups. TLS, SASL, and ACLs protect access to Kafka but do not affect ISR membership.

12

How does min.insync.replicas control Kafka write durability?

min.insync.replicas is the minimum number of in-sync replicas required before Kafka accepts writes that request acknowledgment from all in-sync replicas. It trades write availability for durability by preventing a producer from succeeding when too few replicas are reliably caught up.

Read full answer

Applies toApache Kafka 3.x and 4.x

How It Works

  • The in-sync replica set (ISR) contains the leader and replicas that are sufficiently caught up with the partition’s log.

  • With acks=all (represented as -1 in the protocol), Kafka checks that the ISR contains at least min.insync.replicas replicas.

  • If the ISR is smaller, the produce request fails with NotEnoughReplicasException.

  • The leader counts as one ISR replica, so a value of 2 requires the leader plus at least one caught-up follower.

  • The setting does not change the partition’s replication factor; it defines the minimum acceptable ISR size for protected writes.

  • The broker-level default is 1. A topic can override it with a stronger value such as 2 or 3.

min.insync.replicas is an availability threshold, not the number of acknowledgments the producer waits for. With acks=all, the producer waits for all replicas currently in the ISR to acknowledge the record.

Failure Behavior

Consider a topic with replication factor 3 and min.insync.replicas=2:

Condition Result for acks=all
All three replicas are in the ISR Writes succeed after all current ISR replicas acknowledge
One broker fails, leaving two in the ISR Writes continue, assuming the remaining replicas are healthy
A second replica falls behind or fails ISR falls below 2; writes are rejected
Only the leader remains in the ISR Writes are rejected, protecting durability

If the leader fails, Kafka can elect another in-sync replica as leader. However, this setting does not disable unclean leader election. For critical data, leave unclean.leader.election.enable=false so an out-of-sync replica cannot become leader and lose committed records.

acks=1 and acks=0 do not provide the same all-replica durability guarantee. Use acks=all when min.insync.replicas is intended to enforce the configured durability floor.

Production Trade-offs

Configuration Availability Durability and Operational Risk
RF 3, min ISR 2 Tolerates one broker failure Common production balance; prevents writes with only one replica
RF 3, min ISR 1 More tolerant of degraded replicas Allows acknowledged writes with no in-sync follower copy
RF 2, min ISR 2 Very low write availability during replica loss Requires both replicas to remain caught up, but still has little redundancy
RF 3, min ISR 3 Writes stop after any replica loss Strongest durability threshold; unsuitable for many highly available systems

A higher value reduces the chance of acknowledging data that exists on too few replicas, but it can increase produce errors during broker maintenance, network problems, or replica lag. Place replicas across failure domains and investigate lagging replicas rather than immediately lowering the threshold.

Configuration

Set the topic-level min.insync.replicas property to the desired threshold and use a replication factor that provides adequate redundancy. A replication factor of 3 with min.insync.replicas=2 is a common starting point because it tolerates one broker or replica failure while refusing writes when only the leader remains.

The topic-level property overrides the broker-level default. Review the effective value during capacity planning and after configuration changes because raising it can make an existing topic unavailable during replica lag or broker maintenance.

Operational Guidance

  • Monitor ISR changes: Track under-replicated partitions, out-of-sync replica counts, offline partitions, and producer NotEnoughReplicasException rates.

  • Diagnose before relaxing policy: Check broker health, disk throughput, network latency, partition imbalance, and excessive consumer or producer load.

  • Use full acknowledgment: Configure producers with acks=all and retain idempotent production for duplicate protection.

  • Do not confuse it with exactly-once processing: Kafka’s exactly-once workflows use transactional production and isolation.level=read_committed; Kafka Streams provides this through processing.guarantee=exactly_once. Idempotent producers reduce retry duplicates but do not by themselves make an arbitrary producer-consumer pipeline end-to-end exactly-once.

  • Secure the cluster separately: TLS, SASL, and ACLs protect confidentiality and authorization but do not affect ISR membership or durability.

13

What happens when a Kafka broker fails?

When a Kafka broker fails, its partitions lose their leader, and Kafka elects a new leader from the in-sync replicas so producers and consumers can continue with minimal interruption. Partitions with no in-sync replica become unavailable unless unclean leader election is explicitly enabled.

Read full answer

Applies toApache Kafka 3.7+ (KRaft mode; ZooKeeper mode is legacy)

Leader Election and Partition Availability

  • Each partition has one leader and zero or more followers. Followers that are caught up form the in-sync replica set (ISR).

  • When the leader broker fails, the controller (in KRaft, the active controller from the metadata quorum) selects a new leader from the ISR.

  • Producers and consumers refresh metadata, discover the new leader, and resume. With acks=all and a healthy ISR, committed records are not lost.

  • If the failed broker was the only in-sync replica for a partition, that partition becomes offline until a replica catches up or unclean leader election is enabled.

  • unclean.leader.election.enable defaults to false in modern Kafka. Enabling it can restore availability by promoting an out-of-sync replica, but it can cause data loss and consumer offset resets.

Client and Consumer Group Impact

  • Producers retry failed sends according to retries and delivery.timeout.ms; idempotent producers prevent duplicate writes during retries.

  • Consumer groups whose coordinator was on the failed broker rediscover the coordinator on another broker and rebalance. Temporary commit failures can lead to duplicate processing, so consumers should remain idempotent.

  • Ordering guarantees remain per partition; a new leader continues from the committed log end, protected by leader epochs.

Recovery and Data Consistency

  • When the failed broker restarts, it rejoins as a follower and truncates any divergent tail using the leader epoch, then catches up from the current leader.

  • Replication lag and under-replicated partitions should return to normal before the cluster is considered healthy.

  • If the broker was also a KRaft controller, the metadata quorum elects another active controller; metadata availability depends on quorum health, not one node.

Operational Monitoring and Trade-offs

  • Watch under-replicated partitions, offline partitions, ISR shrink rate, request latency, and broker/controller availability.

  • Durability and availability trade off through replication.factor, min.insync.replicas, acks, and rack awareness. A common production setting is replication factor 3 with min.insync.replicas=2 and acks=all.

  • Avoid treating a restarted broker as healthy until ISR membership and lag metrics recover; premature traffic shifts can increase replication pressure.

Best Practices

  • Keep unclean.leader.election.enable=false unless availability is explicitly prioritized over data integrity.

  • Use rack-aware replica placement so one failure does not remove all replicas for a partition.

  • Alert on sustained under-replication and offline partitions, not just broker process liveness.

  • Test failover in staging with controlled broker termination and verify client retry, rebalance, and recovery behavior.

14

What makes unclean leader election risky in Apache Kafka?

Unclean leader election is risky because it can make a stale replica the partition leader when no in-sync replica is available, trading strict data consistency and potentially losing acknowledged records for faster recovery.

Read full answer

Applies toApache Kafka 3.9 and 4.x in KRaft mode

Failure Mechanism

  • Each Kafka partition has one leader and zero or more follower replicas.

  • The in-sync replica set (ISR) contains replicas that have caught up with the leader’s log. A normal, or clean, leader election selects only an ISR replica.

  • If the leader fails and no ISR replica is available, Kafka normally leaves the partition without a leader and rejects writes until a synchronized replica recovers.

  • With unclean.leader.election.enable=true, Kafka may instead elect an out-of-ISR replica whose log might be behind the former leader.

  • This affects partition leadership, not KRaft controller elections. KRaft controllers require a voting quorum and cannot safely elect a minority controller.

  • When divergent replicas later reconcile, Kafka can truncate log portions that conflict with the elected leader, permanently discarding records.

Data and Delivery Impact

Behavior Clean election Unclean election
Leader candidate ISR replica only Out-of-ISR replica may be selected
Availability Partition may remain unavailable Partition can recover if a stale replica is available
Divergent logs Avoids selecting a known lagging replica Can select and truncate divergent logs
Acknowledged records May still be lost with weaker acknowledgements such as acks=1 Can be lost even when producers use acks=all
  • Producer acknowledgements: acks=all waits for all replicas in the current ISR, but it cannot guarantee durability if those replicas subsequently fail and only a stale replica remains eligible.

  • min.insync.replicas: With acks=all, this setting makes writes fail when too few replicas are synchronized. It does not prohibit an unclean election after a leader failure.

  • Transactions: Kafka’s exactly-once transaction flow can atomically commit a consume-transform-produce operation only while the required records remain in the log. It cannot restore transactional records missing from the newly elected leader.

  • Ordering: Ordering remains defined per partition for records that survive. Unclean election can nevertheless create logical gaps and discard records, so it does not preserve a complete ordered stream.

  • Consumer groups: Consumer offsets are not automatically repaired to reflect the truncated log. Consumers can encounter offset-out-of-range behavior or continue from an offset that no longer corresponds to the same data.

Operational Trade-offs

  • Unclean election prioritizes partition availability over consistency. It may be acceptable in an explicitly lossy recovery scenario, but the organization must accept that acknowledged data can disappear.

  • A typical replicated deployment uses a replication factor of at least three, min.insync.replicas=2, producer acks=all, and idempotent producers. These settings reduce exposure but do not make unclean election safe.

  • Monitor UnderReplicatedPartitions, OfflinePartitions, leader-election activity, broker errors, and controller or broker logs for unclean-election events.

  • During rolling upgrades, upgrade one broker at a time and verify that ISR membership and replica health recover before continuing.

  • Replication is not a backup. Mirror critical topics to an independent cluster and regularly test restoration procedures.

Production Practices

  • Keep unclean.leader.election.enable=false on every broker. This is the default read-only broker setting, so changing it requires a controlled configuration rollout and broker restart.

  • Investigate the original broker crash, network partition, disk failure, or failed upgrade rather than repeatedly forcing leader changes.

  • If an unclean election occurs, pause or rate-limit affected producers, preserve broker and controller logs, compare partition offsets, and identify potentially missing records.

  • Restore adequate ISR capacity and confirm synchronization before declaring the incident resolved. Avoid manually promoting a stale replica unless the same acknowledged-data-loss risk is explicitly accepted.

  • Protect broker, controller, and client connections with TLS and SASL authentication. ACLs limit unauthorized access but do not change the consistency consequences of an unclean election.

15

What does an Apache Kafka producer do?

Kafka producer is a client application that publishes records to Kafka topics. It selects partitions, batches records for efficient transmission, handles broker acknowledgments and retries, and provides delivery behavior according to its reliability configuration.

Read full answer

Applies toApache Kafka 4.x

How It Works

  • ProducerRecord: A producer sends a key, value, and optional headers to a topic. Kafka stores the record in the selected partition.

  • Partition selection: The producer obtains topic metadata, chooses a partition, and sends the request to that partition’s leader. The default partitioner generally maps a key to a consistent partition, while records without keys can be distributed across partitions to reduce hotspots.

  • Batching: The RecordAccumulator groups records into batches. The producer’s sender thread transmits these batches asynchronously, reducing network overhead and increasing throughput.

  • Replication: The leader broker appends the record and replicates it to followers in the partition’s in-sync replica set, or ISR.

  • KRaft: Kafka 4.x supports KRaft rather than ZooKeeper. KRaft controllers maintain the cluster metadata log, while partition leaders handle record writes.

Kafka preserves record order within a partition, not across an entire topic. Increasing a topic’s partition count can change key-to-partition mappings, so applications requiring long-term ordering need an explicit partitioning strategy or a stable event identifier.

Delivery Guarantees

Configuration Acknowledgment behavior Main trade-off
acks=0 The producer does not wait for a broker acknowledgment. Lowest latency, but send failures may not be reported to the application.
acks=1 The leader acknowledges after appending the record to its local log. A leader failure before replication can lose the record.
acks=all The leader waits for all current in-sync replicas, including itself, to acknowledge the record. Stronger durability, but latency depends on ISR health.
  • At-least-once delivery: Use acks=all and allow retries. If a request times out, the producer cannot know whether the broker committed the record; retrying it can therefore create a duplicate.

  • Idempotent delivery: The Kafka 4.x client enables idempotence by default. It prevents duplicate records caused by retries within the same producer session and partition sequence. Setting enable.idempotence=false explicitly disables it; incompatible acks, retries, or max.in.flight.requests.per.connection values are rejected with a configuration error rather than silently overriding idempotence.

  • Exactly-once processing: Idempotence alone does not make an entire application workflow exactly once. Kafka Streams exactly-once processing uses Kafka transactions to atomically publish output records and commit the corresponding consumed offsets. For an equivalent direct-client workflow, write records and offsets through the same transaction and use isolation.level=read_committed to read only committed records.

  • Ordering: Idempotent producers preserve partition order across retries. Without idempotence, multiple in-flight requests combined with retries can reorder records; setting max.in.flight.requests.per.connection=1 avoids that scenario at the cost of throughput.

Performance and Operations

  • Tune batch.size, linger.ms, and compression.type for throughput, while monitoring end-to-end latency rather than maximizing batch size indiscriminately.

  • Reuse a long-lived KafkaProducer; it is thread-safe and maintains metadata, batching, and connection state.

  • delivery.timeout.ms bounds the total time allowed for a send, including retries and waiting for the broker response. request.timeout.ms controls the timeout for an individual broker request.

  • Monitor producer record-error and retry rates, batch size, queue time, request latency, metadata age, and throughput.

  • Monitor brokers for under-replicated partitions, shrinking ISR, disk utilization, request latency, and garbage-collection pressure.

  • Kafka Connect sink connectors use producers internally, while Kafka Streams applications use producers directly, including transactional producers for exactly-once processing.

  • Confluent Schema Registry is a separate service supporting Avro, Protobuf, and JSON Schema. Schema or payload validation occurs only when compatible serializers or connectors are configured to perform it; it is not automatic behavior of the Apache Kafka producer.

Security and Failure Modes

  • Use TLS for encryption in transit and SASL authentication such as SCRAM-SHA-512 for client identity.

  • Grant only required topic-write and transactional-ID permissions through Kafka ACLs. Keep credentials outside source control and inject them from a secret manager.

  • Common incidents include metadata timeouts, unavailable leaders, disk exhaustion, prolonged ISR shrinkage, oversized records, and incompatible schemas.

  • An acks=all producer can still fail when a partition lacks enough healthy replicas. Configure min.insync.replicas, maintain sufficient broker redundancy, and plan capacity accordingly.

Production Best Practices

  • Pre-create topics instead of depending on broker-side automatic topic creation.

  • Keep acks=all, idempotence, and suitable retry limits enabled for important events.

  • Give every concurrently running transactional producer a unique transactional.id; reuse that ID across restarts of the same instance for fencing and recovery, but never share it between active instances.

  • Validate schemas before publication and evolve them compatibly.

  • Treat successful send completion—not merely a successful send() call—as evidence that the broker accepted the record.

16

How does a Kafka producer choose a message's partition?

A Kafka producer selects a message’s partition on the client. With the Kafka 4.x Java client’s built-in partitioner, an explicitly specified partition takes priority; otherwise, a non-null key is mapped using murmur2-compatible hashing, while a null key uses sticky partition assignment.

Read full answer

Applies toApache Kafka 4.x broker, CLI tools, and Java producer client

Partition Selection Rules

Record configuration Built-in result Operational implication
Explicit partition Uses that partition directly If it does not exist, the producer reports an invalid-partition error rather than selecting another partition
Non-null key Hashes the key and applies modulo to the topic’s current partition count The same key routes to the same partition while the partition count remains unchanged and the built-in partitioner is used
Null key Uses sticky assignment for the current producer batch There is no stable key-to-partition relationship across batches
Custom partitioner Uses the implementation configured by partitioner.class when no explicit partition is supplied It can implement business-specific routing, but must be tested against partition growth and failure scenarios

With the built-in partitioner, adding partitions changes the modulo calculation and can remap some keyed records. Historical and new records for the same key may therefore reside in different partitions, breaking assumptions about colocated ordering or compaction.

How Selection Works

  1. The producer retrieves metadata for the topic, including its partitions and leaders.

  2. It selects the partition locally according to the record and the configured partitioner.

  3. It batches records and sends each batch to the selected partition’s leader.

  4. After a leader change or a retryable failure, the producer refreshes metadata and retries the request as permitted by its configuration.

  5. The returned Future or completion callback reports the assigned partition and offset on success, or the delivery error on failure.

A successful call to send() only means that the record was accepted into the producer’s local batching process; it does not prove that the broker durably appended it. Applications should handle future or callback errors and monitor producer error and retry metrics.

Delivery Guarantees and Failure Modes

  • Ordering: Kafka preserves log order within a partition, not across a topic. The built-in partitioner maps a given key consistently, but disabling idempotence while allowing multiple requests in flight can permit retry reordering. Kafka 4.x enables idempotence by default.

  • At-least-once delivery: acks=all, retries, replication, and an appropriate min.insync.replicas configuration provide at-least-once producer behavior, subject to the durability of the replicated cluster. Retries can duplicate records when idempotence is disabled.

  • Idempotent retries: Kafka 4.x defaults enable.idempotence to true. It requires acks=all, a positive retry limit, and max.in.flight.requests.per.connection no greater than 5; it prevents duplicates caused by producer retries within a producer session.

  • Exactly-once processing: Kafka transactions are required for an atomic consume-transform-produce workflow. Use a stable, instance-specific transactional.id and configure consumers with isolation.level=read_committed. External database or HTTP side effects still require idempotency, an outbox, or an equivalent design.

  • Hot partitions: A small number of keys can send disproportionate traffic to one partition. Partition count alone does not correct a hot-key workload.

  • Explicit partitioning: Pinning records to a partition creates coupling to the current partition layout and can concentrate load. Missing partitions cause send failures rather than automatic repartitioning.

Production Practices

  • Inspect the partition count and leaders with kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders.

  • Monitor per-partition traffic, producer errors and retries, request latency, under-replicated partitions, and consumer lag.

  • Treat partition growth as a data-routing operation: it can remap keys and separate records that are expected to be compacted together.

  • Use custom partitioners only when business-specific routing is necessary. Test them with null keys, hot keys, partition growth, leader changes, and broker failures.

  • Protect producer access with TLS or SASL and appropriate ACLs. Partition selection is a routing mechanism, not a security boundary.

17

What are Kafka message keys and why do they matter?

In Apache Kafka, a message key is an optional byte array attached to each record that the producer uses, by default, to choose the destination partition; records sharing the same key are routed to the same partition, which is what provides per-entity ordering and enables log compaction.

Read full answer

Applies toApache Kafka 3.x (KRaft or ZooKeeper-based clusters)

How Partitioning Uses Keys

  • The default partitioner hashes the key (MurmurHash2) and takes the result modulo the partition count, so a given key consistently maps to the same partition as long as the partition count does not change.

  • A null key bypasses hashing; modern clients use a sticky partitioner that batches records to one partition for a while, then rotates, improving throughput for unordered data.

  • You can supply a custom partitioner when routing rules need to differ from key hashing, for example to isolate a tenant or region.

Why Keys Matter

  • Ordering: Kafka guarantees order only within a partition, so keying by an entity ID (user, order, device) keeps that entity’s events in sequence.

  • Locality: a consumer processing a key sees the full history for that entity without cross-partition coordination.

  • Log compaction: compacted topics retain the latest record per key, so keys are mandatory for changelog-style topics and Kafka Streams state stores.

  • Stream processing: Kafka Streams joins, aggregations and windowed operations rely on keys to co-partition and group data correctly.

Key Design Trade-offs

Approach Benefit Risk
Stable high-cardinality key (user ID, order ID) Even distribution, per-entity ordering None significant
Low-cardinality key (country, status) Simple grouping Hot partitions and consumer lag
null key Maximum throughput for unordered events No ordering, no compaction
Composite key Joins across multiple attributes Larger keys, more hashing cost

Operational Considerations

  • Partition skew: a hot key can saturate one partition and its broker; monitor per-partition produce rate and consumer lag.

  • Partition count changes: adding partitions breaks the key-to-partition mapping, so plan partition counts before production traffic.

  • Serialization consistency: producers and consumers must agree on key serialization; a mismatch can create distinct keys or deserialization failures.

  • Compaction limits: a compact-only topic keeps the latest record for each key indefinitely, while older values for that key are removed over time. Tombstones (null values) are kept for delete.retention.ms and then removed; retention.ms/retention.bytes apply only when the topic uses cleanup.policy=delete or compact,delete.

Best Practices

  • Choose immutable, high-cardinality business identifiers as keys.

  • Avoid null keys when ordering or compaction is required.

  • Keep keys small; large keys increase network and storage overhead.

  • Test partition distribution with production-like key cardinality before go-live.

  • Alert on per-partition lag, not just consumer-group lag, to catch hot keys early.

18

What do the Kafka producer acks settings 0, 1 and all mean?

The acks producer setting controls how many broker acknowledgements Kafka waits for before considering a send successful. It is the primary knob trading durability against latency and throughput.

Read full answer

Applies toApache Kafka 3.x and 4.x (KRaft), confluent-kafka clients

How Each Value Behaves

  • acks=0: The producer sends the record and immediately considers it successful without waiting for any broker response. This gives maximum throughput and lowest latency, but the broker may never have received the record; a broker failure or network drop can silently lose data. There is no delivery confirmation, so broker-side loss cannot be detected or recovered through acknowledgements.

  • acks=1: The producer waits for the partition leader to acknowledge after appending the record to its log. This protects against a single failed send being retried, but if the leader crashes before replicating to followers, the record can be lost even though the producer believed it succeeded. It was the default in older clients.

  • acks=all (also accepted as -1): The producer waits until the leader and all in-sync replicas (ISR) acknowledge. This gives the strongest durability guarantee: a record is not acknowledged until it exists on every replica currently in the ISR. Latency is higher because it waits for the slowest in-sync replica.

Comparison

Setting Waits for Durability Latency Typical use
0 Nothing Lowest Lowest Metrics, logs where loss is acceptable
1 Leader only Medium Low General workloads, default in older clients
all Leader + all ISR Highest Highest Financial, audit, critical events

Production Considerations

  • acks=all alone is not enough. Pair it with min.insync.replicas (broker/topic config) so the write fails rather than silently degrading when too few replicas are in sync. A common safe setup is replication factor 3 with min.insync.replicas=2: a write succeeds if at least 2 replicas (leader + one follower) acknowledge, and fails if only the leader is available.

  • Idempotence (enable.idempotence=true) should be enabled with acks=all to prevent duplicates on retries. Java clients 3.0+ default it on when acks=all; set it explicitly in other clients or versions.

  • Ordering: With retries enabled and idempotence off, a failed batch retried after a later batch succeeds can reorder messages. Idempotence plus max.in.flight.requests.per.connection=5 (or 1 without idempotence) preserves per-partition ordering.

  • Exactly-once semantics require acks=all, idempotence, and transactional producers/consumers (transactional.id, isolation.level=read_committed).

  • Monitoring: Watch record-error-rate, request-latency-avg, and ISR shrinkage. A shrinking ISR with acks=all increases latency and can cause producer timeouts; alert on UnderReplicatedPartitions.

Best Practices and Pitfalls

  • Do not use acks=0 for data you cannot afford to lose; there is no delivery confirmation at all.

  • Do not assume acks=1 is durable: a leader crash before replication loses acknowledged data.

  • Always set min.insync.replicas when using acks=all, otherwise a single surviving replica silently weakens the guarantee.

  • Avoid mixing acks=all with very low request.timeout.ms; producers will time out and retry, increasing load.

  • Prefer acks=all as the default for production topics; tune linger.ms and batch.size to recover throughput rather than weakening durability.

19

What does an idempotent Kafka producer guarantee?

An idempotent Kafka producer prevents a retried message from being appended multiple times to the same Kafka partition. It uses a producer ID, epoch, and per-partition sequence number so the broker can recognize retries as duplicates rather than creating duplicate records.

Read full answer

Applies toApache Kafka 3.x and 4.x

How It Works

  • Producer ID (PID): The broker assigns the producer a unique identifier when it initializes the producer session.

  • Epoch: Each producer session receives an epoch, allowing the broker to reject state from an older or fenced producer session.

  • Sequence number: The producer maintains a sequence for each topic partition.

  • Broker deduplication: If a request is retried, the broker compares the PID, epoch, partition, and sequence number. A repeated sequence is acknowledged without appending another record.

  • Safe retries: The producer can retry failed requests without creating duplicates, improving resilience without changing the application’s processing model.

Idempotence is scoped to retries performed by the same producer session. It does not automatically deduplicate two separate calls to send(), nor does it prevent duplication after the producer restarts and receives a new PID.

Required Configuration

Setting Recommended value Reason
enable.idempotence true Enables PID- and sequence-based deduplication. It defaults to true in Kafka 3.x and 4.x, but explicit configuration is clearer.
acks all Required for an idempotent producer and prevents acknowledgement before the record is replicated to the current in-sync replicas.
max.in.flight.requests.per.connection 5 or less Kafka rejects larger values when idempotence is enabled. The default is 5.
retries Retain the default or configure explicitly Retries provide availability during transient failures; idempotence makes those retries safe from duplicate appends.

For stronger durability, configure the topic’s min.insync.replicas appropriately. For example, acks=all with min.insync.replicas=2 prevents writes when fewer than two replicas are in sync, trading availability for consistency.

Guarantees and Limitations

  • Per-partition ordering: Kafka preserves ordering within a partition, not across partitions or topics.

  • Not global exactly-once: Idempotence removes duplicates caused by producer retries. It does not make database writes, HTTP requests, or other external side effects exactly once.

  • Not application deduplication: Sending the same business event twice creates two producer operations and therefore two records. Use a stable event ID or database uniqueness constraint when that must be prevented.

  • Consumer behavior: A consumer that processes a record and crashes before committing its offset can process it again. This is normal at-least-once processing.

  • Exactly-once processing: Use Kafka transactions with a transactional producer and configure consumers with isolation.level=read_committed. External systems still require coordinated idempotency or transaction handling.

Production Failure Modes

  • OutOfOrderSequenceException can occur when broker or client sequence state is corrupted or when requests for the same producer ID and epoch contain an unexpected sequence number.

  • InvalidProducerEpochException can occur when the broker no longer accepts the producer’s epoch, such as when a newer producer instance has fenced the older one.

  • A producer restart may establish a new PID; an application-level retry after that restart can still duplicate an event.

  • Using acks=all can increase latency and reduce write availability during leader elections or replica failures.

  • With transactional producers, a stable transactional.id lets Kafka fence an older producer instance and prevent concurrent transactional writes.

Operational Best Practices

  • Monitor producer record-error-rate and record-retry-rate, along with broker logs for producer-sequence and epoch errors.

  • Alert on under-replicated or offline partitions and on sustained ISR shrinkage.

  • Use TLS with SASL_SSL or another secure transport for production connections.

  • Keep clients and brokers on supported, compatible versions and upgrade brokers before relying on newer producer behavior.

  • Use transactional IDs and initTransactions() only when transaction semantics are required; ordinary idempotence does not require Kafka transactions.

20

How do Kafka producer retries cause duplicates or message reordering?

Producer retries in Apache Kafka resend a batch when an acknowledgment is missing or a transient error occurs. Because the broker may have already persisted the original attempt, retries can create duplicate records, and with multiple in-flight batches they can also break per-partition ordering.

Read full answer

Applies toApache Kafka 3.x (KRaft) with confluent-kafka / Java clients

How Retries Work

  • The producer sends a batch and waits for an acknowledgment according to acks (0, 1, or all).

  • On retriable errors (for example NOT_ENOUGH_REPLICAS, network timeouts) or a missing ack within delivery.timeout.ms, the client retries up to retries, spacing attempts by retry.backoff.ms.

  • If a non-retriable error occurs or the delivery timeout is exceeded, the send fails and the application must decide whether to retry at a higher level.

How Duplicates Arise

  • The broker may successfully append the record but the acknowledgment is lost on the network; the producer retries and the broker appends it again.

  • Application-level retries after a failed send() or flush() can also produce duplicates.

  • An idempotent producer prevents this by attaching a producer ID and per-partition sequence numbers; the broker deduplicates retries with the same sequence number.

How Reordering Arises

  • With max.in.flight.requests.per.connection greater than 1, batches can be in flight concurrently.

  • If batch 1 fails and is retried only after batch 2 succeeds, the final log order can become batch 2 then batch 1.

  • With idempotence enabled, Kafka preserves order for up to 5 in-flight requests. With idempotence disabled, set max.in.flight.requests.per.connection=1 to guarantee order, at a throughput cost.

Key Configuration

Setting Effect
retries Maximum number of send attempts
acks Durability level of each write
enable.idempotence Deduplication and ordering protection
max.in.flight.requests.per.connection Concurrent in-flight batches per connection
delivery.timeout.ms Total time budget for a send including retries

Production Guidance

  • Keep enable.idempotence=True (default in Kafka 3.x clients) for exactly-once-per-partition writes.

  • For end-to-end exactly-once across consume, process, and produce, combine idempotence with transactions.

  • Monitor producer metrics such as record-retry-total, record-error-total, and request latency.

  • Avoid acks=0 when ordering or durability matters, since there is no acknowledgment to trigger a retry.

  • Keep max.in.flight.requests.per.connection at or below 5 when idempotence is enabled.

21

What do batch.size and linger.ms control in Kafka?

batch.size and linger.ms are producer-side settings that control how records are grouped into batches before being sent to a Kafka broker. Together they balance throughput against latency: batch.size caps how large a batch can grow, while linger.ms caps how long the producer waits for more records to fill that batch.

Read full answer

Applies toApache Kafka 3.x Java producer (librdkafka-based clients such as confluent-kafka-python use the same ideas but different defaults: `batch.size=1000000` bytes and `linger.ms=5`; Kafka 4.0 also changed the Java default `linger.ms` to `5`)

How Batching Works

A Kafka producer does not send every record immediately. Instead, it accumulates records for the same partition into an in-memory batch. A batch is sent when any of these happens:

  • The batch reaches batch.size bytes.

  • linger.ms milliseconds have elapsed since the first record was added to the batch.

  • The producer’s send buffer is under memory pressure (buffer.memory).

  • flush() or close is called.

This means linger.ms = 0 (the default) makes the producer send as soon as a batch is available, favoring low latency, while a positive value lets small bursts coalesce into larger requests, improving throughput and compression efficiency.

The Two Settings

  • batch.size: Maximum size in bytes of a single batch per partition. Default is 16384 (16 KB). Records larger than this are sent in their own batch. Raising it helps when individual records are large or throughput is high, but it increases memory use per in-flight batch.

  • linger.ms: Maximum time in milliseconds to wait before sending a partially filled batch. Default is 0. Raising it (for example to 5–20 ms) increases batching and throughput at the cost of added end-to-end latency.

Trade-offs

Setting change Throughput Latency Memory / CPU
Increase batch.size Higher Slightly higher Higher per partition
Increase linger.ms Higher Higher (bounded by value) Lower request rate
Both at defaults Lower Lowest More requests

Production Guidance

  • Tune these together with compression.type (for example lz4 or zstd): larger batches compress better, so raising linger.ms often amplifies compression gains.

  • Do not set linger.ms so high that it exceeds your latency SLO; it is an upper bound, not a fixed delay.

  • Watch producer metrics such as record-send-rate, batch-size-avg, request-latency-avg, and buffer-available-bytes. If batch-size-avg is far below batch.size, batching is ineffective and linger.ms may be too low or traffic too sparse.

  • batch.size is per partition, so a topic with many partitions multiplies memory usage; size buffer.memory accordingly.

  • These settings do not change delivery semantics: with enable.idempotence=true and acks=all, retries remain safe regardless of batching.

Common Pitfalls

  • Setting linger.ms high on a low-traffic topic can add latency without improving batching because batches rarely fill.

  • Oversizing batch.size without increasing buffer.memory can cause BufferExhaustedException under load.

  • Assuming batch.size is a global limit; it is per partition, so total in-flight memory can be much larger.

22

How does message compression work in Kafka producers?

Compression in Kafka producers happens at the batch level: the producer groups records destined for the same partition into a batch, compresses that entire batch with the configured codec, and sends the compressed bytes to the broker, which stores them as-is.

Read full answer

Applies toApache Kafka 3.x (KRaft or ZooKeeper mode) with the Java/Python clients; configuration names are stable across 2.x–3.x.

How It Works

  • Records are accumulated per partition into a RecordBatch (controlled by batch.size and linger.ms).

  • When the batch is sent, the producer compresses the whole batch with the codec named by compression.type.

  • The broker does not decompress by default; it stores and replicates the compressed batch, and consumers decompress on fetch.

  • Because compression is per-batch, a single record or tiny batches gain almost nothing. Batching is what makes compression effective.

Compression Codecs

Codec CPU cost Ratio Typical use
none none 1x Debugging, already-compressed payloads
gzip high best Cold storage, low-throughput, high ratio needs
snappy low moderate General balanced workloads
lz4 very low moderate High-throughput, latency-sensitive pipelines
zstd low–moderate best among fast codecs Recommended for most modern workloads

zstd is usually the best trade-off on Kafka 2.1+ and is a recommended choice for most workloads, but the client default remains none.

Key Configuration

  • compression.type — codec name; default none in many clients, so set it explicitly.

  • batch.size — bytes per partition batch (default 16384); larger batches compress better.

  • linger.ms — wait time to fill a batch (default 0); raising it improves compression and throughput at the cost of latency.

  • compression.type on the broker (default producer) — if set to a specific codec, the broker recompresses incoming batches, which costs broker CPU and is rarely wanted.

Operational Considerations

  • CPU vs. network/disk: compression moves cost to producers and consumers; measure end-to-end latency, not just broker throughput.

  • Idempotence/transactions: compression composes fine with enable.idempotence=true and transactions; the compressed batch is still the unit of retry.

  • Mixed codecs: different producers can use different codecs on the same topic; brokers and consumers handle each batch independently.

  • Monitoring: watch record-send-rate, batch-size-avg, compression-rate-avg, and producer CPU. Low compression-rate-avg with high CPU usually means batches are too small.

  • Upgrades: all modern clients support all codecs; very old consumers may lack zstd/lz4 support, so verify consumer versions before switching.

Best Practices

  • Prefer zstd for new clusters; use lz4 when producer CPU is the bottleneck.

  • Raise linger.ms (5–20 ms) and batch.size together so compression has enough data to work on.

  • Keep payloads uncompressed at the application layer (e.g., don’t gzip JSON before producing) and let Kafka handle it.

  • Avoid forcing recompression on the broker; leave compression.type=producer.

  • Benchmark with realistic record sizes; tiny records may not benefit from any codec.

23

How does a Kafka consumer read and process topic records?

A Kafka consumer is a client that fetches records from assigned topic partitions, processes them, and records its progress through consumer-group offsets. Consumers normally join a consumer group, allowing multiple instances to share partitions and scale horizontally.

Read full answer

Applies toApache Kafka 4.x in KRaft mode, including the Java client and bundled CLI tools

How a Consumer Reads Records

Kafka topics are partitioned logs stored on brokers. A partition leader maintains the authoritative log, while followers replicate it for broker-failure tolerance; KRaft controllers manage cluster metadata and leadership.

After subscribe, polling triggers metadata discovery, group coordination, and partition assignment. Subsequent polls return records in a ConsumerRecords batch from the consumer’s assigned partitions. The application processes those records and then decides whether and when to commit the corresponding group offsets.

Common consumer patterns include:

  • Java or Python clients: Application code calls poll(), processes the returned records, and manages commits.

  • Kafka Connect sink connectors: Consume records from Kafka topics and write them to external systems.

  • Kafka Streams applications: Consume input topics, maintain local state, and produce output topics.

Kafka preserves ordering only within a partition. Records from different partitions can be processed concurrently, and key-based ordering requires all records for that key to remain in one partition.

Groups, Assignments, and Offsets

A group.id identifies a consumer group. The group coordinator assigns each partition to at most one active member; members beyond the available partition count remain idle.

Adding, removing, or restarting a member causes a rebalance, during which partitions can move. The cooperative sticky assignor can reduce unnecessary movement compared with an eager rebalance. Changing a topic’s partition count can remap keyed records and disrupt continuity for a key, so partition counts should normally remain stable.

A committed offset identifies the next record that the group should consume, not the last record physically processed. If the group has no valid committed offset, auto.offset.reset selects a starting position, commonly earliest or latest.

If processing prevents the application from calling poll() within max.poll.interval.ms, the coordinator removes the consumer from the group and reassigns its partitions. Uncommitted records can then be delivered again.

Processing and Delivery Guarantees

A poll cycle consists of obtaining an assignment, fetching records, processing the returned batch, and committing offsets according to the required delivery guarantee.

Processing pattern Result
Process, then commit At least once; a crash before commit can cause duplicate processing
Commit, then process At most once; a crash after commit can lose processing
Atomically publish output records and consumed offsets in one Kafka transaction Exactly once for that Kafka read-process-write flow, when the required transaction settings are used

For at-least-once processing, set enable.auto.commit=false and commit only after successful processing. Handle commit failures and rebalances explicitly, and make business operations idempotent where possible because failures can still produce duplicates.

For Kafka exactly-once processing, the consumer must use isolation.level=read_committed. The producer must use a unique transactional.id, enable idempotence, use acks=all, and place output sends and sendOffsetsToTransaction calls in the same producer transaction. read_committed returns committed transactional records only up to the last stable offset. These guarantees do not cover arbitrary database or API side effects; those require idempotency, an outbox, or another coordination mechanism.

Operational Behavior and Best Practices

  • max.poll.records limits the size of one returned batch. Large batches increase processing latency and the risk of exceeding the poll interval.

  • Keep the interval between poll() calls below max.poll.interval.ms; session.timeout.ms instead controls failure detection through missed heartbeats.

  • Monitor consumer lag by partition, poll latency, commit failures, rebalance frequency, fetch errors, broker disk usage, and under-replicated partitions.

  • Use bounded retries and a dead-letter topic for poison records. Account for duplicate delivery and avoid retries that permanently block later partitions.

  • On compacted topics, cleanup eventually removes obsolete records for the same key, but it is not an immediate update mechanism; consumers can observe multiple versions before compaction. Ordinary delete-policy topics expire records through retention policies.

  • Replication is not a backup. Mirror important topics to another cluster or external storage, and periodically test restoration.

  • When using a Schema Registry, configure the matching deserializer and compatible schemas, and monitor incompatible schema IDs separately from lag.

  • Use TLS and SASL/SCRAM in production, restrict ACLs to required topic and group operations, and never store plaintext credentials in source control.

  • Test rolling broker and consumer upgrades with representative workloads, verify lag recovery and rebalance behavior, and maintain rollback and incident-recovery plans.

24

What is a Kafka consumer group?

A Kafka consumer group is a logical set of consumers that share the same group.id and cooperate to consume subscribed topic partitions. Kafka assigns each subscribed partition to at most one member of a group at a time, while different groups consume the same records independently.

Read full answer

Applies toApache Kafka 3.6 through 4.x (KRaft mode), Confluent Platform 7.5+

How It Works

  • Group membership: Consumers configure the same group.id to join the same group. Each consumer may receive multiple partitions and may remain idle if there are insufficient partitions.

  • Coordination: The broker acting as the group coordinator manages membership, heartbeats, committed offsets, and rebalances. The elected group leader calculates partition assignments using a partition-assignment strategy.

  • Partition ownership: A partition is delivered to only one consumer within a group, preserving partition-level ordering for that group. Other consumer groups maintain separate offsets and receive the records independently.

  • Rebalancing: When a member joins, leaves, times out, or changes its subscription, Kafka revokes and reassigns partitions as necessary. A consumer that stops calling poll() can be removed after max.poll.interval.ms, even if its background heartbeat thread is healthy.

Configuration and Rebalancing

Property Default Purpose
group.id Required Identifier shared by members of one group
session.timeout.ms 45000 Time without a heartbeat before the member is considered failed
heartbeat.interval.ms 3000 Background heartbeat frequency
max.poll.interval.ms 300000 Maximum permitted delay between poll() calls
max.poll.records 500 Maximum records returned by one poll()
enable.auto.commit true Enables automatic offset commits
auto.offset.reset latest Initial position when no valid committed offset exists
partition.assignment.strategy RangeAssignor, CooperativeStickyAssignor Ordered list of supported assignment strategies

The leader uses the first configured assignment strategy supported by every member, so the default list normally selects RangeAssignor. CooperativeStickyAssignor supports incremental rebalancing and should be configured explicitly when its reduced partition revocation is desired.

Offsets and Delivery Semantics

A committed offset identifies the next record to consume and is stored in the internal __consumer_offsets topic.

Mode Requirements and behavior
Automatic commit The default client commits positions during poll() cycles. This does not atomically coordinate processing with commits, so it does not by itself guarantee end-to-end at-least-once or exactly-once behavior.
At-least-once Disable automatic commits and commit after the business side effect succeeds. A crash before the commit causes the records to be processed again, so handlers must be idempotent.
At-most-once Commit before processing. A failure after the commit but before processing loses the record from the consumer’s perspective.
Exactly-once Use a Kafka transaction, a transactional producer with a stable transactional.id, isolation.level=read_committed, and sendOffsetsToTransaction before commitTransaction. This atomically commits output and input offsets within Kafka; it does not make an arbitrary external database transaction atomic.

Producer idempotence prevents duplicate Kafka writes from that producer but is not, by itself, consumer exactly-once processing.

Operational Behavior

  • Lag: Monitor per-partition consumer lag, poll latency, processing latency, rebalance frequency, and commit failures. A growing lag means consumers cannot keep up with production.

  • Failure handling: Broker failover may delay rebalances, while consumer crashes cause partition movement, duplicate processing, and temporary lag spikes.

  • Offset retention: The broker setting offsets.retention.minutes defaults to 10080 minutes, or seven days. This retention applies when the group has no active members; an expired group subsequently initializes from auto.offset.reset.

  • Static membership: A stable, unique group.instance.id can preserve assignment during a brief restart. It does not prevent all rebalances, and concurrently running members must not share the same value.

  • Security: Grant consumers READ access to the required topics and READ access to the consumer group. Restrict group and topic administration separately and use TLS or SASL for non-local connections.

Best Practices

  • Size each group against the partitions available for its subscribed topics; additional consumers remain idle.

  • Use cooperative sticky assignment when reducing rebalance disruption is more important than compatibility with older clients.

  • Keep KafkaConsumer operations on the consumer thread. For long processing, pause affected partitions or use controlled worker processes with explicit pause, offset, and commit coordination.

  • Increase max.poll.interval.ms only when processing legitimately requires it; do not use it to hide stalled applications.

  • Prefer manual commits after durable processing for at-least-once behavior, and design handlers to tolerate duplicates.

25

How does Kafka assign partitions to consumers in a consumer group?

Kafka assigns each partition of a subscribed topic to exactly one consumer within a group, using a group coordinator broker and a pluggable partition assignment strategy. The assignment is recomputed whenever group membership changes, and the result is distributed to all members through the JoinGroup/SyncGroup protocol.

Read full answer

Applies toApache Kafka 3.x (brokers and clients, KRaft or ZooKeeper mode)

How Assignment Works

  • A consumer group has one group coordinator broker, selected by hashing the group ID to a partition of the internal __consumer_offsets topic.

  • All members send heartbeats to the coordinator. When a consumer joins, leaves, or the partition count changes, the coordinator triggers a rebalance.

  • During rebalance, one member (the leader, elected by the coordinator) runs the configured PartitionAssignor and produces an assignment map; the coordinator distributes it via SyncGroup.

  • Each partition is owned by exactly one consumer in the group at any time, which is what gives per-partition ordering guarantees.

Assignment Strategies

Strategy Behavior Best For
RangeAssignor Divides partitions of each topic into contiguous ranges per consumer Single-topic groups; can cause imbalance with multiple topics
RoundRobinAssignor Interleaves partitions across consumers Multi-topic groups; better balance
StickyAssignor Minimizes partition movement on rebalance while keeping balance Reducing rebalance disruption
CooperativeStickyAssignor Incremental rebalancing; only moves partitions that must change Production workloads; avoids stop-the-world pauses

Set the strategy via partition.assignment.strategy on the consumer. In Kafka 3.x, CooperativeStickyAssignor is recommended for production workloads, while the client default remains RangeAssignor.

Eager vs Cooperative Rebalancing

  • Eager (classic): all consumers revoke all partitions, then rejoin. Causes a full stop-the-world pause and reprocessing risk.

  • Cooperative (incremental): consumers only revoke partitions that are moving. Other partitions keep being consumed, reducing downtime. Requires all members to use a cooperative assignor.

Static Membership

Set group.instance.id to a stable value per consumer instance. On restart, the consumer rejoins with the same identity and can reclaim its previous partitions without triggering a full rebalance, which is valuable for rolling deploys and pod restarts in Kubernetes.

Guarantees

  • Ordering is guaranteed only within a partition, not across partitions of a topic.

  • A partition is consumed by one consumer at a time, so offsets advance monotonically per partition.

  • With CooperativeStickyAssignor, partitions that do not move are never revoked, preserving in-flight processing.

Operational Considerations

  • Rebalance storms: frequent consumer restarts or slow processing cause repeated rebalances. Monitor consumer group state and rebalance rate.

  • Consumer lag: track records-lag-max per partition; uneven assignment shows up as skewed lag.

  • Session timeouts: session.timeout.ms and max.poll.interval.ms must exceed worst-case processing time or the consumer is kicked out, triggering rebalance.

  • Partition count changes (adding partitions) force a rebalance and break key-based ordering assumptions for new partitions.

Best Practices

  • Prefer CooperativeStickyAssignor for production.

  • Use static membership (group.instance.id) for stateful or long-running consumers.

  • Keep max.poll.records sized so processing fits within max.poll.interval.ms.

  • Avoid mixing assignor types within one group.

  • Monitor per-partition lag, not just group-level lag, to detect skew.

26

What causes a Kafka consumer rebalance and why is it disruptive?

A consumer rebalance occurs when Kafka recalculates membership or partition assignments for a consumer group. It is disruptive because affected consumers must pause or stop work while assignments change, and uncommitted records may subsequently be processed again.

Read full answer

Applies toApache Kafka 3.9 and 4.0

What Causes a Rebalance?

Kafka’s group coordinator manages each group.id. Common triggers include:

  • A consumer joining or leaving the group.

  • A consumer crashing or failing heartbeat checks.

  • A consumer exceeding the classic protocol’s max.poll.interval.ms without calling poll().

  • The subscribed topic gaining or losing partitions.

  • A consumer changing its subscription.

  • A planned close() that sends a LeaveGroup request.

Protocol behavior differs:

  • Classic consumer group protocol: Membership depends on poll() calls and heartbeats. If a consumer does not poll within max.poll.interval.ms, the coordinator removes it and reassigns its partitions. Assignors include eager strategies, which revoke all partitions before reassignment, and cooperative strategies, which revoke only partitions that must move.

  • New consumer group protocol: Preview in Kafka 3.9 and enabled by default in Kafka 4.0. It uses heartbeat-based membership and server-side assignment, and it does not remove a member merely because it exceeded max.poll.interval.ms.

For the classic protocol, cooperative sticky assignment can be selected with partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor. The client must also use the classic protocol, such as with group.protocol=classic. Built-in assignors consider subscriptions, available partitions and members, and the selected strategy; they do not infer arbitrary application capacity.

Why Rebalancing Disrupts Processing

  • Processing pause: Eager rebalancing revokes every assignment, while cooperative rebalancing affects only moving partitions.

  • Partition handoff: The new owner must fetch from the correct offset, and the previous owner must stop processing revoked partitions.

  • Duplicate processing: If processing completes but the offset commit fails or ownership is lost first, the next owner processes those records again.

  • Uneven load: New members can receive partitions, while existing members wait or receive different assignments.

  • Extra coordination: Group membership, metadata, assignment, and offset state must remain consistent across consumers.

Rebalancing does not inherently lose records that have been successfully committed to Kafka. Its principal risk is replaying work when application processing and offset commits are not atomic.

Offsets, Ordering, and Delivery Semantics

Kafka guarantees record order within a partition, not across partitions. For at-least-once processing:

  1. Poll records.

  2. Process them successfully.

  3. Commit the offset representing the next record to consume.

  4. Ensure commits never advance beyond work that has actually completed.

A ConsumerRebalanceListener can stop work and attempt a final commit in onPartitionsRevoked. Revocation commits are best-effort and may fail because a rebalance is already in progress. In onPartitionsLost, ownership has already been lost, so the consumer must not assume that another commit is possible.

Producer idempotence does not eliminate consumer-side duplicates. Exactly-once processing requires a transactional design, such as Kafka Streams exactly-once semantics with appropriately configured brokers, transactional producers, source-offset commits, and read_committed consumers. External side effects still require idempotency or an outbox-style workflow.

Monitoring and Troubleshooting

Use kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processing to inspect membership, assignments, current offsets, and lag.

During an incident, correlate:

  • Rebalance rate and duration.

  • Consumer lag before and after membership changes.

  • Classic-protocol max.poll.interval.ms violations.

  • Heartbeat failures and network instability.

  • Long-running handlers or blocked calls between poll() calls.

  • Deployments, rolling restarts, topic changes, and autoscaling events.

In the classic protocol, keep max.poll.interval.ms comfortably above worst-case processing time and call poll() before that deadline. A worker pool can isolate expensive work, but it must preserve per-partition ordering and commit only offsets whose records have completed. pause() and resume() can control work flow, but the consumer must still poll regularly.

Production Practices

  • Prefer the new protocol on Kafka 4.0 when all clients support it; otherwise configure cooperative sticky assignment for the classic protocol.

  • Process records sequentially within a partition unless weaker ordering is acceptable.

  • Use manual commits after successful processing for predictable at-least-once behavior.

  • Handle both onPartitionsRevoked and onPartitionsLost, and design processing to tolerate replay.

  • Use rolling restarts and review autoscaling policies to limit simultaneous membership changes.

  • Treat occasional deployment-related rebalances as normal; frequent rebalances accompanied by lag spikes usually indicate slow processing, network instability, or unstable membership.

  • Remember that longer timeouts reduce false removals but delay recovery from genuinely stuck or dead consumers.

27

What is cooperative incremental rebalancing and static membership in Kafka?

Cooperative (incremental) rebalancing and static membership are consumer-group features that reduce rebalance disruption: cooperative rebalancing moves only partitions that must change instead of stopping all consumers, while static membership lets a restarted consumer reclaim its previous assignment without an immediate rebalance.

Read full answer

Applies toApache Kafka 3.x (static membership since 2.3, cooperative rebalancing since 2.4)

How Cooperative Rebalancing Works

  • In an eager rebalance, every consumer revokes all partitions, then receives a new assignment, causing a global processing pause.

  • In cooperative rebalancing, the assignor computes the difference between current and target assignments.

  • Consumers keep partitions that remain assigned to them and only revoke partitions that must move.

  • Reassignment may take multiple rebalance rounds, but unaffected partitions continue processing.

  • All members of the group must use a cooperative assignor, such as CooperativeStickyAssignor.

How Static Membership Works

  • A consumer sets a unique group.instance.id.

  • The broker treats that member as persistent rather than purely session-based.

  • If the consumer disconnects, its assignment is retained for session.timeout.ms.

  • If it rejoins with the same instance ID before that timeout expires, it can resume without triggering a rebalance.

  • If the timeout expires, the member is removed and a rebalance occurs.

Configuration

  • partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor

  • group.instance.id=consumer-instance-1

  • session.timeout.ms=45000

  • heartbeat.interval.ms=15000

  • group.instance.id must be unique per consumer process or pod.

  • A restarted instance should come back with the same ID before session.timeout.ms expires, so it keeps its previous assignment without a rebalance.

Comparison

Feature Cooperative rebalancing Static membership
Main problem reduced Stop-the-world rebalances Rebalances during restarts
Key setting partition.assignment.strategy group.instance.id
Scope Whole consumer group Individual consumer instance
Still rebalances when Membership or partition ownership changes Session timeout expires, or the group’s members or subscriptions change

Production Considerations

  • Use both together for rolling deployments and pod restarts.

  • Keep session.timeout.ms long enough for normal restarts but short enough to detect real failures.

  • Ensure max.poll.interval.ms accommodates worst-case processing time.

  • Monitor consumer lag, rebalance frequency, group member count, and session timeouts.

  • Keep client and broker versions compatible with the chosen rebalance protocol.

Common Pitfalls

  • Mixing eager and cooperative assignors in the same group.

  • Reusing one group.instance.id across multiple running consumers: the coordinator fences the older member, so the two instances keep knocking each other out.

  • Setting session.timeout.ms too low for deployment or GC pauses.

  • Assuming static membership prevents all rebalances; it only delays removal until the session timeout.

28

How does Apache Kafka commit consumer group offsets?

Kafka stores a committed offset for each topic-partition in a consumer group. The broker acting as the group coordinator writes that progress to __consumer_offsets, allowing the group to resume after restarts or rebalances.

Read full answer

Applies toApache Kafka 4.x, Java 17+

How Offsets Are Stored

  • Consumer group: group.id identifies the group whose progress is stored.

  • Coordinator: Consumers send OffsetCommit requests to the broker coordinating their group.

  • Internal topic: The coordinator writes offsets to the replicated, log-compacted __consumer_offsets topic.

  • Offset semantics: A committed offset is the next position to consume in that partition; each partition has an independent offset.

  • Missing offsets: If no valid committed offset exists, auto.offset.reset selects the starting position. Its default is latest; earliest and none are also supported.

  • Ordering: Offsets do not change Kafka’s ordering guarantee, which exists only within a partition. Committing past unfinished work can skip that work.

Automatic and Manual Committing

Mode Configuration Behavior
Automatic enable.auto.commit=true (default) During poll(), the client periodically commits positions returned by an earlier poll.
Manual synchronous enable.auto.commit=false and commitSync() The call blocks until the commit succeeds or fails.
Manual asynchronous enable.auto.commit=false and commitAsync() The commit proceeds asynchronously and can invoke a callback.

For business processing, the usual flow is:

  1. Poll a batch of records.

  2. Complete the durable business operation.

  3. Commit the consumer positions with commitSync().

  4. Continue polling.

If the application fails before committing, the group eventually rebalances and resumes from its previous committed offsets, replaying the uncommitted records. This provides at-least-once processing when combined with manual commits; handlers should still be idempotent because a successful business update followed by a failed offset commit can cause a duplicate.

Asynchronous commits reduce latency, but the application must handle their callbacks and ensure that an in-flight commit is not lost during shutdown.

Exactly-Once Offset Handling

Kafka transactions can atomically publish output records and commit the corresponding consumed offsets:

  • Use an idempotent transactional producer with a unique, stable transactional.id for each logical producer.

  • Begin a transaction and produce the output records.

  • Call producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata()) with the consumed offsets.

  • Commit the transaction.

  • Use isolation.level=read_committed on consumers that must see only committed transactional records.

This does not make arbitrary external side effects exactly once. Database or HTTP operations outside Kafka still require idempotency, deduplication, or an outbox/inbox pattern. Kafka Streams supports its own exactly-once state-store and output guarantees with processing.guarantee=exactly_once; downstream consumers still need read_committed.

Operations and Failure Modes

  • Lag: kafka-consumer-groups.sh --describe compares committed offsets with partition log-end offsets.

  • Rebalances: A consumer that exceeds max.poll.interval.ms can lose its partitions. Continue polling during long work or pause assigned partitions while still meeting the poll interval.

  • Commit failures: Coordinator elections, broker failures, network faults, and authorization errors can fail commits. The client retries retriable failures, but fatal exceptions must terminate or alert the application.

  • Retention: offsets.retention.minutes controls how long offsets for inactive groups are retained. Expired offsets do not delete topic records; the reset strategy determines where consumption resumes.

  • Replication: Monitor __consumer_offsets, under-replicated partitions, and broker health. Replication is not a backup; use an appropriate Kafka backup or replication strategy for business topics.

  • Security: Use TLS and SASL in production. Grant READ on the consumer group and READ and DESCRIBE on the input topic; Kafka group ACLs do not use a DESCRIBE group operation.

Code Example

Manual synchronous committing with the Java client:

import java.time.Duration;
import java.util.List;
import java.util.Map;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;

public class OrderConsumer {
    public static void main(String[] args) {
        Map<String, Object> config = Map.of(
                ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
                "kafka-1.example.com:9092",
                ConsumerConfig.GROUP_ID_CONFIG, "order-processors",
                ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false,
                ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest",
                ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
                StringDeserializer.class.getName(),
                ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
                StringDeserializer.class.getName());

        try (KafkaConsumer<String, String> consumer =
                     new KafkaConsumer<>(config)) {
            consumer.subscribe(List.of("orders"));

            while (true) {
                ConsumerRecords<String, String> records =
                        consumer.poll(Duration.ofSeconds(1));

                for (var record : records) {
                    processOrder(record.key(), record.value());
                }

                if (!records.isEmpty()) {
                    // Commits the next position for consumed assigned partitions.
                    consumer.commitSync();
                }
            }
        }
    }

    private static void processOrder(String orderId, String order) {
        // Replace with an idempotent, durable business operation.
    }
}

Inspect committed offsets and lag:

bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --group order-processors \
  --describe

Show earliest and latest topic offsets with Kafka 4.x syntax:

bin/kafka-get-offsets.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --topic orders earliest

bin/kafka-get-offsets.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --topic orders latest

Best Practices

  • Disable automatic commits when correctness depends on successful business processing.

  • Commit only after the complete business transaction succeeds.

  • Make handlers idempotent because replay and duplicate delivery remain possible.

  • Confine a KafkaConsumer to one thread; it is not thread-safe.

  • Alert on sustained lag, rebalance frequency, commit failures, and under-replicated partitions.

  • Test broker termination and coordinator failover to verify replay behavior.

  • Reset offsets deliberately while the group is inactive, first using --dry-run:

bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --group order-processors \
  --topic orders \
  --reset-offsets \
  --to-earliest \
  --dry-run
29

What is the difference between auto commit and manual commit in Kafka?

In Kafka, offset commits record how far a consumer group has read; auto commit lets the client persist offsets on a timer, while manual commit gives the application explicit control over when offsets are saved.

Read full answer

Applies toApache Kafka 3.x (Java consumer client and confluent-kafka Python).

How Offset Commits Work

  • Each consumer tracks its read position per partition; committing writes that position to the internal __consumer_offsets topic.

  • With auto commit (enable.auto.commit=true), the client commits the offset of the last polled batch on a timer (auto.commit.interval.ms, default 5000 ms), typically during poll().

  • With manual commit, auto commit is disabled and the application calls commitSync()/commitAsync() (Java) or consumer.commit() (Python) only after it has finished processing.

Auto Commit

  • Advantage: minimal code and simple consumption loops.

  • Risk: the committed offset can move ahead of actual processing. A crash after commit but before processing causes at-most-once loss; a crash before commit causes at-least-once duplicates. Semantics are effectively mixed and hard to reason about.

  • Best suited for low-stakes telemetry where occasional loss or duplication is acceptable.

Manual Commit

  • Advantage: you commit only after successful processing, giving clear at-least-once semantics.

  • Trade-offs: more code and responsibility; commitSync() blocks and can stall the poll loop, while commitAsync() is non-blocking but needs error handling and retry on rebalance.

  • Exactly-once requires transactions: sendOffsetsToTransaction() plus a transactional producer and isolation.level=read_committed on consumers.

Comparison

Aspect Auto commit Manual commit
Control Client-driven timer Application-driven
Semantics Mixed loss/duplicate risk Clear at-least-once (or exactly-once with transactions)
Failure handling Limited Full control over retries and ordering
Complexity Low Higher
Production fit Low-stakes streams Critical pipelines

Best Practices and Pitfalls

  • Disable auto commit for any pipeline where correctness matters (enable.auto.commit=false).

  • Commit after processing completes, and handle onPartitionsRevoked to commit in-flight work before a rebalance.

  • Keep processing idempotent so at-least-once redelivery does not corrupt state.

  • Avoid committing per record; batch commits reduce overhead on __consumer_offsets.

  • For exactly-once, use transactional producers and read_committed consumers; never mix non-transactional writes with manual commits expecting exactly-once.

  • Monitor consumer lag and commit latency; stalled commits often indicate a blocked commitSync() or a slow downstream.

30

What happens when a Kafka consumer crashes before committing offsets?

When a Kafka consumer crashes before committing offsets, the messages it already fetched but did not commit are redelivered to the replacement consumer after a rebalance, resulting in at-least-once processing and possible duplicate side effects.

Read full answer

Applies toApache Kafka 3.x (KRaft mode)

Offset Commit Semantics

Kafka stores consumer progress as offsets in the internal __consumer_offsets topic. An offset is only durable progress once it is committed. Fetching a record does not advance the committed offset, so a crash before commit leaves the last committed position unchanged.

Crash Before Commit: What Happens

  • The consumer’s session expires after session.timeout.ms (or it leaves the group explicitly).

  • The group coordinator triggers a rebalance and assigns its partitions to surviving or new consumers.

  • The new owner starts reading from the last committed offset, so uncommitted records are consumed again.

  • Ordering within a partition is preserved, but the same record may be processed multiple times.

Duplicate Processing and Idempotency

Because redelivery is expected, downstream writes must be idempotent. Common patterns include: - Upserts keyed by a business key or Kafka record key. - Deduplication using a processed-event store keyed by topic-partition-offset. - Transactional outbox or database transactions that include the offset.

Auto-Commit vs Manual Commit

  • Auto-commit (enable.auto.commit=true) commits periodically based on auto.commit.interval.ms; a crash can still lose up to that interval of progress and cause duplicates.

  • Manual commit gives control over when progress is recorded, typically after successful processing.

  • Committing before processing gives at-most-once semantics and risks data loss; committing after processing gives at-least-once semantics.

Exactly-Once Considerations

Kafka’s exactly-once semantics apply to consume-transform-produce pipelines using idempotent producers and transactions. A consumer alone cannot make arbitrary external side effects exactly-once; idempotent application logic is still required.

Operational Considerations

  • Monitor consumer lag and rebalance frequency; frequent rebalances can indicate slow processing or unstable consumers.

  • Set max.poll.interval.ms high enough for worst-case processing time to avoid false rebalances.

  • Use a dead-letter topic for poison-pill records so one bad message does not block the partition.

  • Ensure graceful shutdown calls consumer.close() to commit final offsets and leave the group cleanly.

Best Practices

  • Prefer manual commits after successful processing for critical pipelines.

  • Make consumers idempotent and track processed offsets where side effects are not naturally idempotent.

  • Keep processing time below max.poll.interval.ms or use a separate worker thread with care.

  • Alert on rising lag, repeated rebalances, and commit errors.

31

How does Apache Kafka consumer lag work, and how is it monitored?

Consumer lag is the difference between a consumer group’s committed offset and the end offset of a Kafka topic-partition. Monitoring it shows whether consumers are keeping up with producers and helps identify stalled consumers, slow processing, insufficient parallelism, and unevenly distributed work.

Read full answer

Applies toApache Kafka 4.0.x–4.1.x with kafka-exporter 1.x

How Lag Is Calculated

For each topic-partition for which the group has a valid committed offset:

lag = log-end-offset - committed-offset

  • Log-end-offset: The next offset Kafka will reserve for an appended record or batch.

  • Committed-offset: The next offset the group intends to consume.

  • Total group lag: The sum of lag across the group’s partitions with valid committed offsets, normally its assigned partitions.

This is an offset difference, not an exact record count. Retention and compaction can remove records while leaving offset gaps; transactional control batches can also consume offsets without producing ordinary application records. Consequently, lag may overstate the number of records still available for processing. Track record age or a source-timestamp metric alongside lag to understand urgency.

A zero value means the numeric offset positions match, but it does not prove that every relevant business record was processed if offsets were skipped or records were removed. If no valid committed offset exists, treat the lag as unavailable rather than zero.

Inspecting and Monitoring Lag

The Kafka CLI command kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processing reports CURRENT-OFFSET, LOG-END-OFFSET, and LAG for each partition. Use --members to inspect assignments and members, and --state to distinguish an empty group from one with no valid offsets.

The kafka-exporter project exposes:

  • kafka_consumergroup_lag

  • kafka_consumergroup_current_offset

  • kafka_consumergroup_log_end_offset

Prometheus queries such as sum by (consumergroup) (kafka_consumergroup_lag) show total group lag, while max by (consumergroup, topic, partition) (kafka_consumergroup_lag) retains partition-level detail. These metrics do not directly provide the age of the oldest unprocessed record; that requires application or additional telemetry.

Broker JMX metrics remain necessary for CPU, network, disk, ISR changes, request latency, fetch rates, and under-replicated partitions, but broker metrics alone do not provide reliable lag for every consumer group.

Production Failure Modes

  • Consumer crash: Lag continues growing while production continues. A replacement consumer triggers a rebalance, and lag decreases only after processing catches up.

  • Slow downstream processing: Lag grows despite healthy brokers and an active consumer.

  • Hot partition: One partition receives disproportionate traffic or requires expensive processing. Aggregate lag can conceal this even when it falls elsewhere in the group.

  • Producer burst: Lag may temporarily increase and then recover if the consumer has sufficient capacity.

  • Offset expiration: After a group has no active members for offsets.retention.minutes, its committed offsets expire. When a consumer subsequently joins without a valid offset, auto.offset.reset determines the initial position.

  • Retention or compaction gaps: Numeric lag can include offset positions whose records no longer exist or are no longer deliverable.

  • Unsafe offset commits: With enable.auto.commit=true, Kafka can commit a polled position before processing finishes, creating a risk of lost work. Disable auto-commit and commit after successful processing for at-least-once behavior; duplicate external side effects remain possible.

Kafka read-process-write exactly-once processing additionally requires an idempotent transactional producer with a stable transactional.id, consumers using isolation.level=read_committed, and one transaction containing both output records and the consumed offsets through sendOffsetsToTransaction. This does not make arbitrary database or API writes exactly once; those require idempotency or a transactional outbox pattern.

Operational and Security Practices

  • Alert on sustained lag growth rate and duration, not only absolute lag.

  • Monitor every partition to expose hot or unevenly processed partitions.

  • Correlate lag with rebalances, processing latency, consumer errors, and broker metrics.

  • Track backlog age or source-timestamp freshness when timely processing is an SLO.

  • Protect kafka-exporter with TLS and SASL, restrict its Kafka principal to the required group and topic permissions, and secure its HTTP metrics endpoint.

  • Never reset offsets automatically merely because lag is high; first determine whether the cause is retention, a failed consumer, uneven assignments, or insufficient downstream capacity.

32

What do auto.offset.reset earliest and latest mean in Kafka?

auto.offset.reset is the consumer-side fallback used when Kafka finds no valid committed offset for an assigned partition. earliest starts at the partition’s log start offset, while latest starts at its log end offset and normally waits for subsequently published records.

Read full answer

Applies toApache Kafka 3.x and 4.x

Exact Behavior

The reset policy applies when:

  • A consumer group subscribes for the first time and has no committed offset.

  • A committed offset has expired or is otherwise unavailable.

  • The committed offset is outside the partition’s valid range, such as after retention removed the referenced records.

If a valid committed offset exists, Kafka resumes from that offset regardless of auto.offset.reset.

Situation earliest latest
New consumer group Partition’s log start offset Partition’s log end offset
Committed offset expired Partition’s log start offset Partition’s log end offset
Committed offset below retention boundary Partition’s log start offset Partition’s log end offset
Committed offset above partition end Partition’s log start offset Partition’s log end offset
Valid committed offset Resume from committed offset Resume from committed offset

The log end offset is the offset that will be assigned to the next record. It is not the same as the high watermark, which is the last fully replicated or stable offset and can be lower during replication lag. Consequently, latest does not mean “read the last existing message”; it normally causes the consumer to wait for records published after its reset position is established.

Consumer Groups and Delivery Semantics

The policy is evaluated independently for each partition assigned to a consumer. Kafka preserves record order within a partition; changing the reset policy changes only the starting position and does not reorder records.

auto.offset.reset does not provide a delivery guarantee by itself:

  • At least once: Process records before committing their offsets. A crash can cause records to be replayed.

  • At most once: Commit offsets before processing. A crash can cause records to be skipped.

  • Exactly once: Perform the read-process-write operation as one Kafka transaction, including the output records and consumer offsets in the transaction. Use a transactional producer and isolation.level=read_committed on consumers that must not observe aborted transactional records; read_committed alone does not make processing exactly once.

Enabling automatic offset commits does not by itself provide exactly-once processing and may commit a position before the corresponding business operation completes.

Configuration

Set auto.offset.reset on each consumer. The Java client and Python confluent-kafka client both use the property name auto.offset.reset with the values earliest and latest.

For command-line testing, kafka-console-consumer.sh accepts the setting through --consumer-property auto.offset.reset=earliest or --consumer-property auto.offset.reset=latest. A broker’s server.properties does not provide a default for this consumer-specific setting.

The policy only takes effect when no valid committed offset exists. Using a group that already has committed offsets continues from those offsets even if the configured policy is earliest.

Production Trade-offs

  • earliest: Useful for replaying retained history, rebuilding projections, or auditing events. It can create substantial consumer lag and broker load.

  • latest: Useful for services interested only in new events. It avoids replaying old data, but records published before the reset position is established are skipped.

  • Consumption gap: latest can leave an application missing records when a new group starts or its previous offset becomes invalid. The records remain in Kafka; the loss is at the application-processing level.

  • Duplicate processing: When earliest resets a group whose prior progress was lost, previously processed records can be replayed. Downstream operations should therefore be idempotent unless transactional exactly-once processing is used.

  • Retention and compaction: Kafka can reset only to offsets Kafka still retains. For compacted topics, offsets may point to gaps because older records have been removed or compacted away.

  • Access control: Reset behavior does not bypass Kafka ACLs. The consumer still needs appropriate permissions to read the topic and join or use the consumer group.

Monitoring and Incident Checks

Use kafka-consumer-groups.sh with the --describe option to inspect each partition’s current offset, log end offset, and lag. During incidents, check:

  • Whether the group has a valid committed offset.

  • Whether topic retention or offsets.retention.minutes invalidated that offset.

  • Consumer lag and rebalance frequency.

  • Partition availability and under-replicated partitions.

  • Whether latest was unintentionally configured for a group that requires complete history.

33

How do you scale Kafka consumers, and what limits the parallelism?

Kafka consumer parallelism is bounded by partition assignment: within one consumer group, each partition is assigned to at most one consumer at a time. The useful ceiling is therefore the number of partitions the group can consume, so consumers beyond that ceiling remain idle while a rebalance assigns them no work.

Read full answer

Applies toApache Kafka 4.x (KRaft), Java 17+, and `kafka-clients` 4.x

Assignment and Limits

  • Partition ownership: A consumer can own multiple partitions, but two members of the same group cannot own the same partition simultaneously. Broker replication protects partition data; it does not multiply consumer-group parallelism.

  • Ordering: Kafka preserves record order within a partition, not across partitions. Ordering for a key therefore requires a stable key-to-partition mapping. Increasing a topic’s partition count can remap existing keys under the default partitioner and disrupt per-key processing order.

  • Independent groups: Each consumer group receives its own assignment, so separate billing and analytics groups can consume every partition in parallel.

Scaling Kafka Consumers

  1. Add partitions to increase the assignment ceiling. Partition counts cannot be reduced.

  2. Add consumer instances with the same group.id, up to the number of relevant partitions across their subscriptions.

  3. Parallelize within a partition when necessary, using bounded executors. Preserve ordering for related keys and commit only after a contiguous prefix has completed.

  4. Tune fetching and polling: max.poll.records, fetch.max.bytes, and max.partition.fetch.bytes affect batch size and memory, but a small max.poll.records value does not permanently cap throughput.

  5. Protect downstream systems: Kafka consumption should not outrun database connections, HTTP limits, or processing capacity.

Limit Operational effect Mitigation
Relevant partitions Hard ceiling on useful group members Add partitions or use separate workloads
Hot partition or key One partition becomes a latency bottleneck Change key distribution or partitioning
max.poll.interval.ms Long processing causes member eviction Process faster, keep polling, or raise the limit
Assignment strategy Eager assignors revoke all partitions before reassignment Prefer CooperativeStickyAssignor
Offset granularity Offsets are committed per partition, not per record Commit the next offset after a contiguous prefix completes
Broker, network, or downstream capacity Fetching or processing slows despite idle consumers Add capacity, use quotas, batches, and backpressure

Delivery Semantics

  • At-least-once: Disable auto-commit and commit offsets after successful processing. A crash or rebalance can replay records, so handlers should be idempotent.

  • Exactly-once: For consume-transform-produce workflows, use a transactional producer with enable.idempotence=true and a transactional.id, and set consumers to isolation.level=read_committed. Output records and consumed offsets must participate in the transaction.

  • Kafka Streams uses processing.guarantee=exactly_once_v2 (the older exactly_once value was removed in Kafka 4.0). Kafka Connect source connectors can run exactly-once when the workers set exactly.once.source.support=enabled and the connector supports it (exactly.once.support=required); this is not a general guarantee for every connector type.

Code Example

# Inspect the partition count.
kafka-topics.sh --bootstrap-server kafka.example.com:9092 \
  --describe --topic orders

# Increase the partition count; choose a value greater than the current count.
kafka-topics.sh --bootstrap-server kafka.example.com:9092 \
  --alter --topic orders --partitions 24

# Inspect assignment and lag.
kafka-consumer-groups.sh --bootstrap-server kafka.example.com:9092 \
  --describe --group billing-consumer
import java.time.Duration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;

public class BillingConsumer {
  public static void main(String[] args) {
    Properties props = new Properties();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
        "kafka.example.com:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing-consumer");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
        StringDeserializer.class.getName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
        StringDeserializer.class.getName());
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
    props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
    props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
        "org.apache.kafka.clients.consumer.CooperativeStickyAssignor");

    Map<TopicPartition, OffsetAndMetadata> pending = new HashMap<>();

    try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
      consumer.subscribe(List.of("orders"));

      while (true) {
        ConsumerRecords<String, String> records =
            consumer.poll(Duration.ofMillis(500));

        for (ConsumerRecord<String, String> record : records) {
          process(record);
          TopicPartition partition =
              new TopicPartition(record.topic(), record.partition());
          pending.put(partition,
              new OffsetAndMetadata(record.offset() + 1L));
        }

        if (!pending.isEmpty()) {
          consumer.commitSync(pending);
          pending.clear();
        }
      }
    }
  }

  static void process(ConsumerRecord<String, String> record) {
    // Idempotent business logic.
  }
}

Production Operations

  • Monitor per-partition lag, rebalance rate, poll latency, and commit failures. Alert on sustained lag growth rather than an isolated spike.

  • Rebalances occur when members join or leave, assignments change, or processing exceeds the relevant timeout. A paused consumer must continue calling poll().

  • Static membership uses a stable, unique group.instance.id per instance. It can preserve assignment across restarts, but duplicate IDs cause fencing and must be avoided.

  • Kafka Connect’s task count is capped by the connector’s tasks.max setting; the connector may create fewer tasks (for example, a sink gains nothing from more tasks than topic partitions), and worker capacity limits how many run well. Kafka Streams tasks derive from the topology and input partitions; num.stream.threads controls how many threads execute them.

  • Retention and compaction affect storage and replay behavior but not the consumer-group assignment ceiling. Schema Registry can add serialization and validation overhead but does not increase parallelism.

  • Use TLS plus SASL for client authentication and grant least-privilege topic and consumer-group ACLs. Never place credentials in source code or container images.

34

What do Kafka’s at-most-once, at-least-once, and exactly-once semantics guarantee?

Kafka’s delivery semantics describe the trade-off between record loss and duplicate processing. Exactly-once processing is available within Kafka’s transactional boundaries, but it does not automatically make arbitrary external side effects exactly once.

Read full answer

Applies toApache Kafka 4.0 clients and brokers (KRaft)

Guarantees

Semantic What it guarantees What can go wrong
At-most-once A record is not deliberately processed more than once by the consumer. The consumer commits its offset before processing the record. A failure after the offset commit but before processing causes data loss. A producer that does not retry can also lose records before they are durably stored.
At-least-once Records are not intentionally discarded. The consumer processes the record before committing its offset. A failure after processing but before the offset commit causes the record to be delivered again, so duplicate processing is possible.
Exactly-once Kafka atomically commits the produced output records and the consumed offsets as one transaction. Aborted transactional records are not exposed to read_committed consumers. The guarantee applies to Kafka records and offsets. External database writes, HTTP calls, and filesystem operations are not covered unless they participate in the same transactional system.

Producer and Consumer Responsibilities

  • Producer acknowledgements and retries: For at-most-once behavior, a typical configuration is acks=0 with retries=0; the producer accepts that a record can be lost without a broker acknowledgement. For at-least-once behavior, use durable acknowledgements such as acks=all, enable retries, and normally enable enable.idempotence=true to avoid duplicates caused by producer retries.

  • Offset commit ordering: At-most-once consumers commit before processing. At-least-once consumers commit after successful processing. Automatic commits are generally unsuitable when the application needs precise control over this ordering.

  • Idempotent processing: At-least-once consumers should make processing effectively idempotent using a business identifier, deduplication table, or equivalent mechanism. Kafka producer idempotence does not prevent a consumer from processing the same business event twice after a crash.

  • Ordering: Kafka preserves order within a partition, not across an entire topic. Related records that require ordering must use the same partition key.

Exactly-Once Requirements and Boundaries

A Kafka consume-transform-produce workflow needs a transactional producer, a stable and unique transactional.id, and explicit transaction handling. The producer writes output records and submits the consumed offsets with its transactional offset-commit API, then commits the transaction. The consumer must disable automatic offset commits and use isolation.level=read_committed.

read_committed consumers do not advance past the last stable offset while an earlier transaction remains open, and they do not expose aborted transactional records. A transaction timeout, producer failure, rebalance, or fencing error can cause the transaction to abort and the input records to be redelivered.

Kafka Streams can provide exactly-once processing for supported Kafka-to-Kafka topologies with processing.guarantee=exactly_once. Kafka Connect support is connector-specific and also requires the worker and connectors to be configured for transactional processing. Schema Registry does not provide delivery guarantees.

Operational Implications

  • Monitoring: Track consumer lag, the last stable offset versus the log end offset, producer errors, transaction aborts, commit failures, under-replicated partitions, and rebalance frequency.

  • Incident handling: Treat transaction aborts and consumer fencing as retryable workflow events. Ensure the application can safely reprocess records after a restart or coordinator change.

  • Retention and compaction: Retention limits replay time, while log compaction removes obsolete records. Neither changes the delivery guarantee or prevents an application from losing its offset or deduplication state.

  • Recovery: Replication protects against broker failure, but not accidental topic deletion or incorrect application writes. Mirror critical topics where appropriate and test restoration procedures.

  • Security: Use TLS, SASL, and least-privilege ACLs for producers, consumers, administrators, and transactional workflows.

Choosing a Semantic

Use at-most-once only when loss is acceptable and duplicates are unacceptable. Use at-least-once for most durable workflows, combined with idempotent processing. Use exactly-once when all relevant state is Kafka records or offsets and the operational cost of transactions is justified; use an outbox pattern or another coordinated mechanism when side effects occur outside Kafka.

35

How do Kafka transactions provide exactly-once processing?

Kafka transactions provide exactly-once processing by atomically committing output records and consumer offsets to the same transaction. A failed transaction is aborted, so read_committed consumers never observe its records; a successful transaction exposes both outputs and offsets together.

Read full answer

Applies toApache Kafka 4.x in KRaft mode

How Exactly-Once Processing Works

A transactional producer uses a unique transactional.id to coordinate writes with Kafka’s transaction coordinator:

  1. initTransactions() initializes the producer and fences any older producer using the same ID.

  2. beginTransaction() starts a transaction.

  3. The producer writes output records.

  4. sendOffsetsToTransaction() adds the consumed input offsets to that transaction.

  5. commitTransaction() atomically makes the outputs and offsets visible.

  6. If processing fails, abortTransaction() discards the output and offset changes.

If the producer crashes before commit, Kafka eventually aborts the transaction after its timeout. Because the input offset was not committed, the input is processed again, but the previous uncommitted output remains invisible. This prevents duplicate visible Kafka records.

Consumers using isolation.level=read_committed stop at the last stable offset while a transaction is open and ignore records from aborted transactions.

Required Configuration

Component Configuration Purpose
Producer enable.idempotence=true Deduplicates retries within the producer session
Producer acks=all Requires all in-sync replicas to acknowledge writes
Producer max.in.flight.requests.per.connection<=5 Preserves ordering with idempotence enabled
Producer transactional.id=<unique-instance-id> Identifies and fences a transactional producer
Consumer isolation.level=read_committed Excludes aborted and uncommitted records
Consumer enable.auto.commit=false Supports manually committing offsets with sendOffsetsToTransaction()
Kafka Streams processing.guarantee=exactly_once_v2 Uses Kafka 4.x exactly-once processing

The producer’s transaction.timeout.ms must not exceed the broker’s transaction.max.timeout.ms. Transaction metadata is stored in the broker’s internal transaction-state topic, whose replication and minimum ISR settings should be configured like other critical internal topics.

Scope and Ecosystem Integration

  • Kafka Streams: exactly_once_v2 combines transactional output writes, offset management, and state-store changelog processing. It provides exactly-once behavior for Kafka topics and Kafka Streams state stores.

  • Kafka Connect: Exactly-once support depends on the individual source and sink connectors. Do not assume it is enabled merely because the cluster supports transactions; connectors must explicitly support the required semantics.

  • Schema Registry: It does not participate in the transaction. Schemas should still be registered and compatibility-checked before production writes. Schema incompatibility can cause records to fail independently of transaction handling.

  • External systems: Kafka transactions do not include databases, HTTP calls, or other side effects. Use idempotent APIs, deduplication, or an transactional outbox when the target cannot participate in the Kafka transaction.

Production Failure Modes

  • Producer fencing: Starting a new producer with an existing transactional.id prevents the previous instance from committing. Use a distinct ID for each concurrently running instance.

  • Timeout or crash: An open transaction eventually terminates. Monitor transaction timeout, abort rate, producer errors, and consumer lag.

  • Long processing: Keep processing shorter than the transaction timeout and consumer poll interval.

  • Broker failures: Spread brokers across failure domains and size replication so a quorum remains available.

  • Partial visibility: Transactions provide atomic visibility in Kafka, not global ordering across partitions or topics.

Inspect consumer progress with:

bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --describe \
  --group orders-processor

Code Example

This Kafka Streams application atomically writes transformed records while managing its consumed offsets:

import java.util.Locale;
import java.util.Properties;

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.KeyValue;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.kstream.Consumed;

public class OrdersProcessor {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "orders-processor");
        props.put(StreamsConfig.CLIENT_ID_CONFIG, "orders-processor-instance-1");
        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.example.com:9092");
        props.put(StreamsConfig.DEFAULT_KEY_SERIALIZER_CLASS_CONFIG,
                Serdes.String().serializer().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERIALIZER_CLASS_CONFIG,
                Serdes.String().serializer().getClass());
        props.put(StreamsConfig.PROCESSING_GUAREE_CONFIG, "exactly_once_v2");
        props.put(StreamsConfig.STATE_DIR_CONFIG, "/var/lib/kafka-streams/orders");

        StreamsBuilder builder = new StreamsBuilder();
        builder.stream("orders", Consumed.with(Serdes.String(), Serdes.String()))
                .map((key, value) ->
                        KeyValue.pair(key, value.toUpperCase(Locale.ROOT)))
                .to("orders-uppercase");

        Topology topology = builder.build();
        KafkaStreams streams = new KafkaStreams(topology, props);
        Runtime.getRuntime().addShutdownHook(new Thread(streams::close));
        streams.start();
    }
}

Best Practices and Pitfalls

  • Give every active producer instance a unique transactional.id; reuse it only when restarting the same logical instance.

  • Keep transaction and poll intervals comfortably above normal processing latency.

  • Use TLS and SASL for broker connections, and restrict topic, consumer-group, and transactional producer ACLs appropriately.

  • Monitor transaction commit rate, abort rate, timeout frequency, broker errors, and consumer lag.

  • Do not use read_uncommitted in applications that require exactly-once visibility.

  • Treat exactly-once as a guarantee within the Kafka boundary, not as a guarantee for arbitrary external side effects.

36

How does the transactional outbox pattern work with Kafka?

The transactional outbox pattern decouples a database commit from event publication by writing business state and an event record to the database in one transaction. A relay later publishes pending outbox records to Kafka, providing at-least-once delivery in the normal success path.

Read full answer

Applies toApache Kafka 4.x and PostgreSQL 14+; the pattern itself is version-independent

How It Works

  1. The application starts a database transaction.

  2. It updates the business table and inserts an event into an outbox table containing an event ID, aggregate key, topic, payload, creation time, and publication state.

  3. It commits the transaction, making the business change and event atomic within PostgreSQL.

  4. A relay claims pending rows and publishes them to Kafka, normally using the aggregate ID as the message key.

  5. After Kafka acknowledges delivery, the relay marks the rows as published. Cleanup later removes or archives them according to policy.

  6. If the relay crashes after Kafka acknowledges but before the database commit, the rows remain pending and are published again.

Kafka brokers store the topic’s partitions and replicas. Replication and acks=all provide Kafka-side durability, subject to the configured minimum in-sync replica count.

Events with the same key normally enter the same partition while the partition count remains unchanged. Kafka preserves producer order within that partition, but not across partitions or aggregates. Multiple relay instances must not publish different events for the same aggregate concurrently if ordering matters.

The outbox does not create an atomic transaction between PostgreSQL and Kafka. A Kafka transaction cannot make the database commit and Kafka publication one unit.

Delivery Guarantees

Approach Failure behavior Guarantee
Separate database and Kafka writes Either write can succeed independently Inconsistent state is possible
Transactional outbox relay A crash after acknowledgement can cause republication At-least-once delivery
Idempotent consumer Duplicates may arrive, but a recorded event ID can suppress repeated business effects Once-only business effect, not once-only delivery
Kafka transactions Produced records and consumed offsets can commit atomically Exactly-once Kafka read-process-write when fully configured

Producer idempotence prevents duplicates caused by retries within that producer session; it does not prevent the relay from publishing the same outbox row again after a crash.

Consumers should commit their offsets only after processing. For Kafka-to-Kafka read-process-write flows, exactly-once processing requires a transactional producer with a stable, instance-specific transactional.id, inclusion of consumed offsets in the transaction, and consumers using isolation.level=read_committed. External database side effects still require an idempotency strategy. Kafka Streams with processing.guarantee=exactly_once can protect Kafka state and output records, but not arbitrary external side effects.

Relay and Topic Design

  • Topics: Pre-create topics with enough partitions for throughput and ordering requirements. An example is three replicas with min.insync.replicas=2, requiring at least three suitable brokers.

  • Claiming rows: Use a short claim transaction or leased ownership fields. If using FOR UPDATE SKIP LOCKED, do not hold database locks during slow Kafka network calls, and preserve per-aggregate ordering.

  • Publishing: Enable idempotence and acks=all; also bound delivery waits and retry behavior so an unavailable Kafka cluster cannot stall relays indefinitely.

  • Schema validation: A schema registry can validate Avro, Protobuf, or JSON Schema payloads and enforce compatibility before publication.

  • Kafka Connect: A CDC connector can replace a custom relay when it captures only the outbox table and preserves the event ID and aggregate key. It does not remove the need for consumer idempotency.

Production Operations

  • Monitoring: Track oldest unpublished-event age, pending-row count, publish latency and errors, consumer lag, under-replicated partitions, and offline replicas.

  • Retention: Clean published outbox rows according to an audit or recovery policy. Kafka topic retention and outbox cleanup are independent.

  • Compaction: Compaction is intended mainly for keyed state or changelog topics. It is not a substitute for consumer deduplication because a duplicate may be delivered before compaction removes it.

  • Poison events: Quarantine malformed or permanently rejected events with their event IDs rather than blocking the entire relay indefinitely.

  • Recovery: Test database restores and Kafka disaster recovery. Replication is not a backup, and uncoordinated restores can produce gaps or duplicates.

  • Upgrades: Kafka 4.x uses KRaft-only metadata operation; do not mix ZooKeeper and KRaft configuration.

Security and Best Practices

  • Use TLS and SASL authentication for Kafka, restrict topics with ACLs, and grant the relay write access only to required topics.

  • Use separate topics for events with different confidentiality or ownership requirements; headers alone do not provide isolation.

  • Protect database credentials, use least-privilege access to the outbox table, and encrypt sensitive payloads before publication.

  • Validate schemas and payload contents at the relay boundary, and never place credentials or unnecessary personal data in Kafka headers.

  • Keep event IDs stable across retries and store processed event IDs in the same database transaction as the consumer’s business effect when exactly-once business processing is required.

37

How does Apache Kafka guarantee message ordering within a partition?

Kafka guarantees partition-level ordering: records appended to the same partition have increasing offsets, and a consumer observes that partition in offset order. It does not guarantee relative ordering across partitions, because different partitions can be processed concurrently.

Read full answer

Applies toApache Kafka 4.x brokers and Java client (`kafka-clients`) 4.x

Ordering Scope and Architecture

  • A topic contains one or more partitions. Each record receives an offset within its partition, and consumers read a partition’s records in increasing offset order.

  • Records in different partitions have no guaranteed relative order. For example, a record in partition 1 may be delivered before one in partition 0.

  • With the default partitioner, records with the same key normally enter the same partition while the partition count remains unchanged. Explicit partition selection gives producers direct control, while a hot key can concentrate load on one partition.

  • Replicas maintain the same log order. After a clean leader failover, committed records remain available; insufficient replication can lose unacknowledged records. Unclean leader election adds data-loss risk and should remain disabled for critical workloads.

Producer-Side Requirements

For reliable ordering with retries or broker failures:

  • Idempotence: enable.idempotence=true prevents duplicate writes caused by retries and preserves order for records sent to the same partition. It defaults to true in the Kafka 4.x Java client; explicitly setting it to false disables it, while enabling it requires acks=all and max.in.flight.requests.per.connection<=5.

  • Durability: acks=all waits for all current in-sync replicas to acknowledge the record. Configure min.insync.replicas to define the minimum acceptable ISR size.

  • In-flight requests: Keep max.in.flight.requests.per.connection no greater than 5 when idempotence is enabled.

  • Ordering context: Multiple producer instances can interleave writes into one partition, so application-level sequencing may still be necessary.

acks and replication settings primarily affect durability; they do not establish ordering between partitions.

Consumer-Side Requirements

Within a consumer group, each partition is assigned to one consumer at a time, and that consumer receives records in offset order. Concurrent processing can nevertheless complete application side effects out of order.

  • Process one record at a time where downstream ordering matters, or explicitly preserve sequence numbers when parallelizing.

  • Set enable.auto.commit=false when processing and offset commits must be coordinated.

  • For at-least-once processing, complete the side effect before committing its offset. A crash between those operations causes replay, so side effects must be idempotent or use an idempotency key.

  • A rebalance can move a partition to another consumer. Rebalance handling must therefore avoid overlapping processing and incomplete side effects.

Delivery Guarantees

Guarantee Typical behavior Ordering implication
At-most-once Offset is committed before processing Records can be lost, with minimal replay
At-least-once Processing occurs before offset commit Records can be duplicated; sequential processing preserves delivery order
Exactly-once within Kafka A transaction commits output records and consumed offsets together Aborted transactional records are excluded by read_committed consumers

Exactly-once processing requires a transactional producer with a transactional.id, coordinated offset submission such as the Java API sendOffsetsToTransaction, and commonly isolation.level=read_committed. read_committed excludes aborted transactional records but still returns committed and non-transactional records. It does not make an external HTTP request or database write exactly-once; use an idempotent side effect, transactional outbox, or equivalent integration.

Production Failure Modes

  • Retries without idempotence: Can create duplicates and, with multiple in-flight requests, reorder records relative to their original send order.

  • Concurrent consumer processing: Kafka’s read order remains intact, but application side effects can complete out of order.

  • Partition count: Too few partitions limits parallelism and concentrates hot keys; too many increase metadata, replication, and recovery overhead.

  • Under-replicated partitions: Can increase producer latency and errors, especially with acks=all and restrictive min.insync.replicas.

  • Consumer lag: A slow partition delays later records; monitor lag per partition.

For example:

kafka-consumer-groups.sh --bootstrap-server broker.example.com:9092 --describe --group order-service

Operational and Security Trade-offs

Ordering requires serialization at some point: producers must preserve their send order for a partition, and consumers must preserve processing order where side effects depend on it. KRaft controllers manage cluster metadata but do not provide cross-partition ordering.

  • Retention and compaction: Retention removes old segments, while compaction keeps the latest value for a key subject to tombstone handling. Both can leave offset gaps, but surviving records remain ordered by offset.

  • Kafka Connect: Source and sink tasks can process partitions independently, so Connect does not create global ordering. Connectors requiring per-key sequence must preserve key affinity and destination ordering.

  • Kafka Streams: A task processes records from each input partition in order; repartition topics create new partition boundaries, and parallelism does not imply global ordering.

  • Schema Registry: Schema IDs and compatibility checks do not participate in broker sequencing.

Production deployments should combine these guarantees with TLS, SASL authentication, least-privilege topic ACLs, partition-level lag monitoring, under-replicated and offline partition alerts, producer error tracking, and tested rebalance handling. Test supported upgrade paths and client compatibility before rolling upgrades.

38

How do you handle poison messages with Kafka dead-letter topics?

A poison message is a record that repeatedly fails processing because it is malformed, incompatible with a schema, or invalid under business rules. Route it to a dead-letter topic (DLT) after bounded retries, while ensuring the original source offset is committed only after the failure has been durably recorded.

Read full answer

Applies toApache Kafka 4.x; confluent-kafka Python 2.x

Failure-Handling Strategy

  • Classify failures as transient or permanent. Retry transient errors with bounded exponential backoff; send permanent failures directly to the DLT.

  • Set enable.auto.commit=false and manually commit offsets only after successful processing or confirmed DLT publication.

  • If DLT publication fails, stop or pause the affected partition and do not commit its source offset. Kafka will redeliver the record after recovery or consumer restart.

  • Keep retry loops bounded. Blocking a partition for too long can trigger max.poll.interval.ms evictions and create a consumer-group rebalance storm.

  • Include the original topic, partition, offset, timestamp, error class, error message, and relevant headers in the DLT record.

DLT Design

A DLT is a normal replicated Kafka topic. Create it with the same partition count as the source topic so preserving the record key also preserves source-partition ordering:

bin/kafka-topics.sh --bootstrap-server kafka.example.com:9092 \
  --create --topic payments.dlt \
  --partitions 12 \
  --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms=1209600000

Replace 12 with the source partition count. For records without a key, explicitly derive a DLT key from the source partition if ordering matters.

For Kafka Connect sink connectors, the relevant settings are:

  • errors.tolerance=all

  • errors.deadletterqueue.topic.name=payments.dlt

  • errors.deadletterqueue.context.headers.enable=true

Inspect committed DLT records with:

bin/kafka-console-consumer.sh \
  --bootstrap-server kafka.example.com:9092 \
  --topic payments.dlt \
  --from-beginning \
  --isolation-level read_committed

Delivery Guarantees and Ordering

Pattern Guarantee
Process, then manually commit the source offset At least once; failures can redeliver the record
Publish to DLT, then commit the source offset At least once; a crash can create duplicate DLT records
Publish to DLT and commit the offset in one Kafka transaction Exactly once for the Kafka DLT record and source offset

Use enable.idempotence=true and acks=all for the DLT producer. For transactions, configure a unique transactional.id per producer instance and consume the DLT with isolation.level=read_committed.

Ordering is guaranteed only within a partition. To retain source-partition order, process each partition serially, preserve the key, and use matching partition counts. Deduplicate nontransactional DLT records using the tuple (original-topic, original-partition, original-offset).

Kafka Streams provides DeserializationExceptionHandler; processing.guarantee=exactly_once_v2 makes Kafka output and offsets atomic, but does not include arbitrary external side effects.

Code Example

import sys
from confluent_kafka import Consumer, Producer, TopicPartition

class PoisonMessage(Exception):
    pass

def process(value):
    if not value:
        raise PoisonMessage("empty or invalid payload")

bootstrap = sys.argv[1]
instance_id = sys.argv[2]

consumer = Consumer({
    "bootstrap.servers": bootstrap,
    "group.id": "payments-workers",
    "enable.auto.commit": False,
    "isolation.level": "read_committed",
})

producer = Producer({
    "bootstrap.servers": bootstrap,
    "transactional.id": f"payments-dlt-{instance_id}",
    "enable.idempotence": True,
    "acks": "all",
})
producer.init_transactions()
consumer.subscribe(["payments"])

delivery_errors = []

def delivered(err, _message):
    if err is not None:
        delivery_errors.append(err)

while True:
    for message in consumer.consume(num_messages=100, timeout=1.0):
        if message.error():
            raise RuntimeError(message.error())

        try:
            process(message.value())
        except PoisonMessage as error:
            producer.begin_transaction()
            try:
                producer.produce(
                    "payments.dlt",
                    key=message.key(),
                    value=message.value(),
                    headers={
                        "original-topic": message.topic().encode(),
                        "original-partition": str(message.partition()).encode(),
                        "original-offset": str(message.offset()).encode(),
                        "error-class": type(error).__name__.encode(),
                        "error-message": str(error).encode(),
                    },
                    on_delivery=delivered,
                )
                producer.flush()

                if delivery_errors:
                    raise RuntimeError(delivery_errors[0])

                # Atomically commit the DLT record and source offset.
                producer.send_offsets_to_transaction(
                    [TopicPartition(
                        message.topic(),
                        message.partition(),
                        message.offset() + 1,
                    )],
                    consumer.consumer_group_metadata(),
                )
                producer.commit_transaction()
            except Exception:
                producer.abort_transaction()
                raise
        else:
            consumer.commit(
                offsets=[TopicPartition(
                    message.topic(),
                    message.partition(),
                    message.offset() + 1,
                )],
                asynchronous=False,
            )

Operations and Security

  • Monitor DLT production rate, DLT consumer lag, oldest unprocessed record age, retry rate, delivery failures, and source-consumer lag. Alert on sustained DLT growth or publication failures.

  • Set DLT retention according to investigation and replay requirements. Replication is not a backup; use appropriate Kafka backup, restore, and disaster-recovery procedures.

  • Restrict source-read, DLT-write, and DLT-read permissions separately. Use TLS and SASL, and restrict network access to approved infrastructure CIDRs such as 10.0.0.0/16.

  • DLT payloads and error headers may contain sensitive data. Minimize payloads, sanitize exception messages, encrypt topics where required, and apply retention or deletion policies.

  • Replay DLT records through a dedicated, validated workflow rather than blindly producing them back to the source topic.

39

How do you implement retries for failed Kafka message processing?

Use bounded retries with retry topics and a dead-letter topic, and acknowledge the source offset only after processing or retry publication succeeds. For consume-process-produce flows, a Kafka transaction can atomically commit the produced record and source offset, but it cannot guarantee exactly-once effects in an external database or API.

Read full answer

Applies toApache Kafka 4.x; confluent-kafka Python 2.x

Retry Pattern

Kafka brokers store partitioned, replicated logs; consumer groups divide partitions among consumers. Kafka has no consumer-side setting that automatically retries application processing errors, so the application must implement the retry policy.

  1. Consume with enable.auto.commit=false.

  2. Retry transient failures locally with exponential backoff and a strict attempt limit.

  3. After exhausting local retries, publish the original payload to a retry topic.

  4. Retry consumers process that topic, optionally delaying records until the time in an x-retry-at header.

  5. Route permanent failures or exhausted retry attempts to a dead-letter topic.

  6. Commit the source offset only after successful handling or successful publication to the retry/DLT topic.

Create retry and DLT topics with the same partition count as the source topic so the original partition can be preserved:

bin/kafka-topics.sh --bootstrap-server 10.0.0.10:9092 --create --if-not-exists --topic orders --partitions 12 --replication-factor 3 --config min.insync.replicas=2

bin/kafka-topics.sh --bootstrap-server 10.0.0.10:9092 --create --if-not-exists --topic orders.retry --partitions 12 --replication-factor 3 --config min.insync.replicas=2 --config cleanup.policy=delete --config retention.ms=86400000

bin/kafka-topics.sh --bootstrap-server 10.0.0.10:9092 --create --if-not-exists --topic orders.dlt --partitions 12 --replication-factor 3 --config min.insync.replicas=2 --config cleanup.policy=delete --config retention.ms=2592000000

Delivery Guarantees and Ordering

Approach Behaviour
At-most-once Commit before processing. A crash loses the message; normally unsuitable.
At-least-once Process first and commit afterward. A crash can cause redelivery, so handlers must be idempotent.
Exactly-once within Kafka Use isolation.level=read_committed plus a transactional producer for output records and consumed offsets.

Kafka guarantees ordering within a partition, not across partitions or topics. A failed record can allow later records from the same partition to succeed unless the application pauses that partition. Also keep processing and backoff below max.poll.interval.ms; otherwise the consumer is removed from the group and another consumer may process the same partition concurrently.

Producer enable.idempotence=true with acks=all prevents duplicates caused by producer retries. It does not make the consumer’s business handler idempotent. External side effects require a unique event ID, a database uniqueness constraint, or an outbox/inbox pattern.

Kafka Platform Integrations

  • Kafka Connect: Sink connectors support errors.retry.timeout, errors.retry.delay.max.ms, errors.tolerance, and errors.deadletterqueue.topic.name. Enable errors.deadletterqueue.context.headers.enable to retain source context.

  • Kafka Streams: Use processing.guarantee=exactly_once_v2 for Kafka state and output transactions. Application-level retry topics or error handlers are still needed for custom processing failures.

  • Schema Registry: Keep the original payload schema when copying records to retry and DLT topics. Store retry metadata in headers rather than changing the payload schema, and verify compatibility before deployment.

Code Example

This transactional Python consumer retries transient failures locally, then moves exhausted or permanent failures to retry/DLT topics:

import json
import time

from confluent_kafka import Consumer, KafkaError, KafkaException, Producer, TopicPartition

SOURCE_TOPIC = "orders"
RETRY_TOPIC = "orders.retry"
DLT_TOPIC = "orders.dlt"
MAX_LOCAL_ATTEMPTS = 3


class TransientError(Exception):
    pass


class PermanentError(Exception):
    pass


def process_event(raw):
    try:
        event = json.loads(raw)
    except (json.JSONDecodeError, UnicodeDecodeError) as exc:
        raise PermanentError("invalid JSON") from exc

    if not event.get("order_id"):
        raise PermanentError("missing order_id")

    return event["order_id"]


def commit_or_route(consumer, producer, message, destination=None, error=None):
    producer.begin_transaction()
    try:
        if destination:
            headers = list(message.headers() or [])
            headers.extend([
                ("x-source-topic", message.topic().encode()),
                ("x-source-partition", str(message.partition()).encode()),
                ("x-source-offset", str(message.offset()).encode()),
                ("x-retry-at", str(int(time.time() * 1000) + 60000).encode()),
                ("x-error", str(error or "processing failed").encode()[:512]),
            ])

            producer.produce(
                destination,
                key=message.key(),
                value=message.value(),
                headers=headers,
                partition=message.partition(),
            )
            producer.poll(0)

        offsets = [
            TopicPartition(message.topic(), message.partition(), message.offset() + 1)
        ]
        producer.send_offsets_to_transaction(
            offsets, consumer.consumer_group_metadata()
        )
        producer.commit_transaction()
    except Exception:
        producer.abort_transaction()
        raise


consumer = Consumer({
    "bootstrap.servers": "10.0.0.10:9092",
    "group.id": "order-workers",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",
    "max.poll.interval.ms": 300000,
})

producer = Producer({
    "bootstrap.servers": "10.0.0.10:9092",
    "transactional.id": "order-worker-1",  # Unique among concurrently running workers
    "acks": "all",
    "transaction.timeout.ms": 60000,
})

producer.init_transactions()
consumer.subscribe([SOURCE_TOPIC])

try:
    while True:
        message = consumer.poll(1.0)
        if message is None:
            continue
        if message.error():
            if message.error().code() != KafkaError._PARTITION_EOF:
                raise KafkaException(message.error())
            continue

        for attempt in range(1, MAX_LOCAL_ATTEMPTS + 1):
            try:
                process_event(message.value())
                commit_or_route(consumer, producer, message)
                break
            except TransientError as exc:
                if attempt == MAX_LOCAL_ATTEMPTS:
                    commit_or_route(
                        consumer, producer, message, RETRY_TOPIC, exc
                    )
                    break
                time.sleep(0.1 * (2 ** (attempt - 1)))
            except PermanentError as exc:
                commit_or_route(consumer, producer, message, DLT_TOPIC, exc)
                break
finally:
    consumer.close()

The transactional.id must be unique among active producer instances. A transaction protects Kafka records and offsets, but an external API call that succeeds before a failed transaction can still be repeated.

Production Operations

  • Monitor source, retry, and DLT lag with kafka-consumer-groups.sh --describe; alert on retry growth, consumer rebalances, transaction timeouts, and growing DLT volume.

  • Pause or isolate a partition when strict ordering matters; otherwise later records may overtake a failed record.

  • Cap retry attempts and DLT retention to prevent infinite poison-message loops and unbounded storage.

  • Size brokers for retry spikes and configure transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2 on a production KRaft cluster.

  • Protect brokers and topics with TLS, SASL, and least-privilege ACLs. Redact sensitive payloads from error headers and logs.

  • Replay DLT records through a controlled tool that preserves the original event ID, audit trail, and duplicate protection.

40

How do Kafka’s time and size retention settings work?

Retention determines how long Kafka retains records in a partition. Kafka supports time-based retention, size-based retention, or both, and deletion happens at whole-segment granularity rather than per record.

Read full answer

Applies toApache Kafka 4.x (KRaft)

How Retention Works

Kafka stores each topic partition as an append-only log divided into segments. With the default cleanup.policy=delete, the log-cleaner thread periodically identifies segments that exceed the configured retention thresholds.

  • Retention is evaluated independently for each partition, not for the topic as a whole.

  • Kafka deletes complete segments, not individual records.

  • The active segment is never deleted. After it rolls, it can become eligible for deletion.

  • log.retention.check.interval.ms controls how frequently retention checks run and defaults to 300000 milliseconds.

  • Replicas perform cleanup independently, so disk usage can temporarily differ between replicas.

Time-Based Retention

Time-based retention uses the largest timestamp recorded in a segment. A rolled segment becomes eligible when its age exceeds the effective retention period.

Setting Scope and behavior
log.retention.ms Broker-level limit in milliseconds; defaults to -1.
log.retention.hours Broker-level limit in hours; defaults to 168, or seven days.
retention.ms Topic-level millisecond override.
retention.hours Topic-level hour override.

A non-negative millisecond value takes precedence over an hour value. Otherwise, a non-negative hour value is converted to milliseconds; if neither applicable value is non-negative, time retention is unlimited. Topic-level values, when present, take precedence over inherited broker values.

Retention is approximate because it operates on rolled segments. For example, an active segment can remain beyond seven days if it has not rolled, while an older rolled segment may be deleted immediately once it exceeds the threshold.

Size-Based Retention

Size-based retention limits the log data retained per partition.

  • log.retention.bytes=-1 means unlimited broker-level size retention.

  • An explicit topic-level retention.bytes=-1 means unlimited size retention for that topic.

  • log.segment.bytes controls segment size and defaults to 1073741824 bytes, or 1 GiB.

  • Kafka deletes the oldest eligible segments until the partition is below the target where possible.

  • The active segment and segment-size granularity can cause actual usage to exceed the configured limit temporarily.

With cleanup.policy=delete, either the time or size threshold can independently trigger deletion.

Code Example

Inspect the effective topic configuration and then configure seven-day time retention plus a 10 GiB per-partition limit:

# Display the topic's retention configuration
bin/kafka-topics.sh \
  --bootstrap-server broker.example.com:9093 \
  --describe \
  --topic orders \
  --config

# Set time and size retention for the topic
bin/kafka-configs.sh \
  --bootstrap-server broker.example.com:9093 \
  --entity-type topics \
  --entity-name orders \
  --alter \
  --add-config retention.ms=604800000,retention.bytes=10737418240

Capacity planning must multiply the per-partition limit by the partition count and replication factor, then allow for indexes, other topics, replicas, and deletion lag.

Operational and Failure Considerations

  • Consumer recovery: If a committed offset is older than the available log start offset, Kafka applies auto.offset.reset. Its default is latest; none causes the consumer to fail with an offset-out-of-range error.

  • Offset expiry: offsets.retention.minutes controls how long offsets for inactive consumer groups are retained and is separate from record retention.

  • Capacity pressure: Retention checks and file deletion are delayed operations, so disks require additional headroom.

  • Timestamp errors: Kafka defaults to producer CreateTime timestamps. A future timestamp can make a segment appear young and extend retention. LogAppendTime uses broker-generated timestamps instead.

  • Secure erasure: Kafka deletes segment files, but disks, SSDs, snapshots, or backups may retain recoverable bytes. Use encryption, restricted access, short retention, and appropriate storage erasure controls for sensitive data.

  • Monitoring: Track broker disk utilization, partition growth, log-start offsets, consumer lag, compaction and deletion metrics, and consumer-group rebalances.

Compaction Versus Deletion

cleanup.policy=compact retains the latest non-null record for each key rather than deleting data because of age or size. Tombstones represent deleted keys and are retained according to delete.retention.ms before the cleaner removes them.

Kafka also supports cleanup.policy=compact,delete. In that combined mode, a segment is deleted only when it is eligible under both compaction and deletion; satisfying only the time or size threshold is insufficient.

41

How does log compaction work in Apache Kafka, and when should it be used?

Log compaction is Kafka’s mechanism for retaining the latest record for each key in a topic while removing records superseded by newer values with that key. It is useful for state-like data where intermediate values are obsolete, but it is not appropriate when every historical event must remain available.

Read full answer

Applies toApache Kafka 3.x and 4.x

How Compaction Works

Kafka does not immediately remove an old value when a newer value with the same key arrives. The broker’s log cleaner periodically scans eligible log segments, identifies obsolete records, and rewrites the affected data into cleaned segments.

  • Key requirement: Records must have the same non-null key to supersede one another. Kafka routes a key to one partition, preserving its ordering within that partition.

  • Cleanup policy: A topic normally enables compaction with cleanup.policy=compact.

  • Asynchronous cleanup: Compaction does not immediately change what consumers observe. A consumer can still read an old value and its replacement before the cleaner removes the obsolete record.

  • Null keys: Records with a null key are not compacted because Kafka cannot associate them with a key whose latest value can be retained.

  • Offset gaps: Compaction removes records without renumbering the remaining offsets, so consumers will encounter gaps.

  • Offset preservation: Consumer-group offsets are independent of compaction. Compacting a topic does not reset committed consumer positions.

The cleaner generally does not compact the active segment. Settings such as min.cleanable.dirty.ratio, min.compaction.lag.ms, and segment.ms affect how quickly dirty segments become eligible for cleaning, but compaction remains asynchronous and should not be treated as immediate key-value storage.

Tombstones and Retention

A tombstone is a record with a non-null key and a null value. Consumers should interpret it as deletion of that key’s current value.

Kafka retains tombstones after compaction for the period defined by delete.retention.ms; it then removes the tombstone itself. Until the tombstone is removed, the key remains represented in the log. A topic can also combine compaction with a delete policy, in which case normal segment, time, or size-based retention can remove additional data.

When to Use It

Compaction is appropriate when only the latest state or metadata value matters:

  • Kafka Streams application state and compacted changelog topics

  • Latest-value configuration or feature-flag records

  • Rebuildable materialized views where intermediate updates are unnecessary

  • Long-lived deduplication markers, provided the application accounts for duplicates before compaction occurs

For example, repeatedly writing the latest customer address under key customer-123 allows Kafka to retain the newest address without preserving every previous address.

Suitable for compaction Better without compaction
Latest state per key Complete event history
Kafka Streams changelogs Audit and compliance logs
Current metadata snapshots Financial transaction ledgers
Rebuildable state Data requiring historical reconstruction

Compaction should not be used as an archival deletion mechanism. It removes superseded values, but it does not provide precise deletion of every record associated with a subject, including data already retained in backups or in other partitions. Privacy workflows need an explicit deletion and retention design.

Guarantees and Non-Guarantees

Kafka provides ordering per partition, not global ordering across a topic. Compaction preserves that model because all records for one key share a partition.

Compaction itself provides neither at-least-once nor exactly-once processing guarantees:

  • Consumers can replay records after a restart or rebalance, including values that have already been processed.

  • Producer idempotence suppresses duplicate sends caused by retries within a producer session, but it does not prevent an application from deliberately publishing the same logical update multiple times.

  • Kafka transactions can provide atomic read-process-write behavior when the application is designed to use them.

  • Kafka Streams exactly-once processing is enabled with processing.guarantee=exactly_once; it requires a functioning broker transaction-state configuration and appropriately sized transaction-state settings.

  • Consumers of transactional topics that must not expose aborted transactions use isolation.level=read_committed.

These are application-processing guarantees, not guarantees created by compaction.

Production Best Practices

  • Use a reasonably small segment.ms or frequent segment rolls when faster compaction is important, while accounting for the additional segment and file-handle overhead.

  • Avoid an excessively aggressive min.cleanable.dirty.ratio; cleaning consumes broker CPU and disk I/O and can affect foreground traffic.

  • Monitor topic disk usage, log-cleaner metrics and backlog, compaction lag, under-replicated partitions, consumer lag, and tombstone handling.

  • Size partitions for expected key cardinality and throughput. Compaction reduces retained records but does not reduce the configured partition count.

  • Use TLS or SASL and topic-level ACLs where access control and encryption are required; compaction provides neither.

  • Use replication factor 3 across sufficiently independent brokers when the workload can tolerate that level of redundancy.

  • Test backups and restores carefully: a compacted backup represents retained state, not the topic’s complete historical event stream.

42

What are Kafka tombstone messages?

A tombstone message is a Kafka record with a non-null key and a null value. In a compacted topic, it represents a deletion, allowing Kafka’s log cleaner to remove older records for that key while preserving the key’s deleted state.

Read full answer

Applies toApache Kafka 3.9 and 4.x

Purpose and Wire Format

A producer sends a tombstone by using a valid key and a null value:

  • Key: Identifies the record being deleted and must not be null.

  • Value: Is null, signaling deletion.

  • Offset: Advances normally like any other Kafka record.

  • Partition: Follows the producer’s configured partitioning behavior; an explicitly assigned partition takes precedence over key-based partitioning.

Kafka does not immediately remove a key from a topic. The tombstone is appended to the log, consumed by clients, and eventually handled by log compaction.

A null key is different: null-keyed records are excluded from compaction and do not act as deletion markers.

Compaction and Retention

Tombstones are primarily useful with topics configured for compaction:

  • cleanup.policy=compact: Compacts records and removes superseded values.

  • cleanup.policy=compact,delete: Compacts records and also applies normal time- or size-based log retention.

  • delete.retention.ms: Controls how long delete tombstones are retained; its default is 24 hours in Apache Kafka 3.9 and 4.x.

During compaction, Kafka keeps the latest non-null value for each key. When it encounters a tombstone, it can discard earlier values for that key. The tombstone remains temporarily so consumers have an opportunity to observe the deletion.

If cleanup.policy does not include compact, Kafka treats the null-valued record as an ordinary record. It is not compacted away as a deletion marker, although an application may still interpret its null value as a delete.

Consumer Behavior

A consumer receives the tombstone as a normal record whose deserialized value is null, provided its deserializer can represent null.

The consumer’s position advances when the record is polled. The committed consumer-group offset follows the consumer’s commit configuration and does not necessarily mean processing has completed; with auto-commit enabled, an offset can be committed before the corresponding record has been successfully processed.

Application behavior is important:

  • A compacted key-value consumer should treat null as “delete this key.”

  • A consumer that ignores null values may retain obsolete state in an external database or cache.

  • Consumers must support null deserialization; schemas and serializers must preserve the tombstone rather than rejecting or dropping it.

  • Kafka Streams uses tombstones in changelogs to signal removals from state stores.

Tombstones should be processed in the same ordered stream as normal updates. Ordering is guaranteed only within a partition.

Delivery and Failure Semantics

A tombstone has the same delivery guarantees as the producer configuration used to send it:

  • With enable.idempotence=true and acks=all, a producer avoids duplicate writes caused by retries.

  • At-least-once delivery can cause the same tombstone to be delivered more than once, so deletion operations should be idempotent.

  • Kafka’s transactional exactly-once model requires a transactional producer and consumers using isolation.level=read_committed; Kafka Streams additionally uses processing.guarantee=exactly_once.

  • A tombstone does not itself provide exactly-once delivery.

If a producer crashes before the broker acknowledges the record, the deletion may not have occurred. Monitoring producer errors and retry metrics is therefore as important as monitoring normal records.

Production Trade-offs and Failure Modes

Tombstones reduce storage for update-heavy compacted topics, but they introduce operational considerations:

  • Expired tombstones: If a consumer falls behind for longer than delete.retention.ms, it may miss the deletion marker. A later replay cannot rely on an expired tombstone to communicate that deletion and may need a current snapshot or changelog.

  • Ignoring nulls: Consumers that silently discard null values can maintain stale application state.

  • Compaction lag: Disk usage can remain high when the cleaner cannot keep up with write volume or segment configuration is unsuitable.

  • Schema compatibility: Avro, JSON Schema, and Protobuf consumers must be able to represent and process null values. Schema Registry rules should account for tombstone-compatible schemas.

  • Security: Protect compacted topics with appropriate Kafka ACLs, TLS, and authentication. A tombstone is a write to the topic, so write authorization is required.

Monitor consumer lag, tombstone processing failures, compaction throughput, segment size, dirty ratio, and disk utilization. During upgrades or incident recovery, verify that consumers can replay null values correctly and that state can be rebuilt safely from a current snapshot or the compacted log.

For example, configure compaction with kafka-configs.sh --bootstrap-server kafka.example.com:9092 --entity-type topics --entity-name customer-state --alter --add-config cleanup.policy=compact,delete, then verify it with kafka-configs.sh --bootstrap-server kafka.example.com:9092 --entity-type topics --entity-name customer-state --describe.

43

How do you choose partition counts for Kafka topics?

Choose a topic’s partition count from expected throughput, required consumer concurrency, key distribution, and operational limits. Partitions are the unit of Kafka parallelism, and each partition is an independently replicated, ordered log.

Read full answer

Applies toApache Kafka 4.x

How Partitions Affect the System

  • Producer parallelism: A producer can send records to multiple partitions concurrently. More partitions can increase aggregate throughput, but only if producers distribute data across them.

  • Consumer parallelism: Within one consumer group, at most one consumer receives records from a partition at a time. A group with more consumers than partitions leaves extra consumers idle.

  • Ordering: Kafka preserves order within a partition. For same-key ordering across producer retries, use an idempotent producer with enable.idempotence=true; otherwise, set max.in.flight.requests.per.connection=1. Partition ordering alone does not preserve producer send order in a non-idempotent retry scenario.

  • Replication: The replication factor is separate from partition count. For example, 12 partitions with replication factor 3 require 36 partition replicas across the cluster.

  • Failure handling: Each partition has one leader and follower replicas. A broker failure can affect several replicas hosted on it; leader election promotes an eligible in-sync follower. If the in-sync replica set falls below min.insync.replicas, production with acks=all can block.

Decision Criteria

Factor How it affects partition count
Throughput Benchmark realistic record size, compression, acknowledgements, and replication. Add partitions only if broker and network capacity remain sufficient.
Consumer concurrency Use approximately the maximum number of active consumers required by each consumer group. Different groups can each consume every partition.
Ordering requirements All records requiring strict mutual ordering must use the same partition, normally by using the same key.
Key cardinality A few very hot keys can overload one partition regardless of the total partition count.
Retention and compaction More partitions mean more log directories, segment files, leader elections, and replica operations.
Recovery time Excessively large topics can lengthen reassignment and recovery operations because partitions move independently.

A useful starting estimate is the greater of the required consumer concurrency and the throughput-derived partition count. Validate it with production-like load tests rather than relying only on a records-per-second calculation.

Avoid creating thousands of nearly empty partitions for a low-volume topic. Metadata, file descriptors, replica scheduling, and reassignment work all grow with partition count. Conversely, one heavily loaded partition can become a throughput bottleneck.

Creating and Increasing Partitions

# Create a topic with 12 independently replicated partitions.
bin/kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9093 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000

# Increase the partition count when additional throughput or consumers are needed.
bin/kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9093 \
  --alter \
  --topic orders \
  --partitions 24

Kafka supports increasing a topic’s partition count, but not decreasing it. With Kafka’s default partitioner, increasing the count changes the hash-to-partition mapping. Existing records remain in their original partitions, while new records for an existing key can move to another partition, so ordering across the increase is not preserved. If that ordering matters, stop and flush producers before the alteration or migrate keyed data through a versioned topic; draining consumers alone cannot restore ordering.

Production Failure Modes

  • Hot partition: One key or partition receives disproportionate traffic. Inspect per-partition byte and request rates; distributing keys imperfectly may not fix an underlying hot-key design.

  • Under-replicated partitions: Insufficient brokers, failed disks, throttling, or slow replicas can leave the in-sync replica set below the configured size.

  • Consumer lag: More partitions do not help if the group already has more consumers than partitions or if processing is slower than ingestion.

  • Delivery semantics: acks=all waits for acknowledgment from all currently in-sync replicas, subject to min.insync.replicas, but does not provide exactly-once delivery by itself. At-least-once consumers should use enable.auto.commit=false and commit offsets after successful processing, while accepting that crashes can cause duplicates. Kafka exactly-once processing requires transactional producers, an atomic consume-transform-produce flow, and consumers using isolation.level=read_committed; it does not make arbitrary external side effects exactly once.

  • Rebalance pressure: Adding partitions can trigger consumer group rebalances. Use suitable session.timeout.ms and max.poll.interval.ms values and make processing idempotent.

Operational Practices

  • Start conservatively, benchmark, and increase partitions before individual brokers or network paths saturate.

  • Record partition count, replication factor, expected maximum consumers, and keying strategy in the topic’s runbook.

  • Monitor broker capacity, network throughput, request latency, under-replicated partitions, offline partitions, partition-size distribution, consumer lag, and rebalance frequency.

  • Keep replication factor independent from partition sizing. A common production baseline is replication factor 3 and min.insync.replicas=2, provided the cluster can tolerate losing one broker.

  • Apply topic-level ACLs, TLS, and SASL appropriately. Kafka authorization is normally managed at topic, group, and cluster-resource levels rather than independently per partition.

  • For compacted topics, validate tombstone handling and ensure consumers have reached the required end offset; deletion reduces storage but does not immediately remove every segment file.

44

How can the partition count of an Apache Kafka topic be reduced?

No—partition count reduction is not supported for an existing Apache Kafka topic. The only supported in-place change to the partition count is increasing it; reducing it requires recreating the topic or migrating to a new topic with fewer partitions.

Read full answer

Applies toApache Kafka 4.x, including KRaft mode

Why Reduction Is Unsupported

A topic’s partition count is part of its metadata and defines separate ordered logs. Reducing that count would require merging partition logs, rewriting partition and offset mappings, and reassigning consumer responsibilities. Kafka intentionally avoids this destructive, error-prone operation.

The usual alteration command only increases partitions:

bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --alter \
  --topic orders \
  --partitions 12

Requesting fewer partitions than the topic currently has fails. kafka-reassign-partitions.sh cannot help either: it reassigns partition replicas between brokers but cannot reduce the number of partitions.

Migration Options

Approach Data handling Offset impact Operational impact
Delete and recreate with the same name Data must be backed up and restored or producers must republish it Old offsets may point beyond restored records Usually requires downtime
Create a topic with a new name Backfill, dual-write, or stream records between topics New topic has separate offsets Preferred for controlled production migrations
Increase the partition count No reduction occurs Consumers rebalance Supported natively, but partitioning and key distribution change

For production, creating a new topic is usually safer. Migrate records, verify completeness, switch consumers, and retain the old topic temporarily for rollback and audit. Kafka replication protects against broker failures but is not a backup by itself.

Code Example

Run these commands from the Kafka installation directory. The following assumes the original orders topic has 12 partitions and the replacement should have 3:

# Inspect the current partition count and configuration
bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe \
  --topic orders
# This cannot reduce an existing 12-partition topic to 3
bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --alter \
  --topic orders \
  --partitions 3
# Create a replacement topic with fewer partitions
# This requires at least three Kafka brokers for replication factor 3
bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --create \
  --topic orders-v2 \
  --partitions 3 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete

Migrate records with a controlled producer/consumer process or during a dual-write phase. After validating the new topic, update producers and consumers to use orders-v2.

After cutover and the required retention period, the old topic can be deleted if delete.topic.enable is enabled:

bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --delete \
  --topic orders

Topic deletion is asynchronous, so verify that the topic is gone through the monitoring system.

Production and Security Considerations

  • Consumer offsets: Group offsets are associated with a topic name and partition. Recreating the same topic name with fewer partitions can leave stale offsets, causing consumers to skip records or start at invalid positions. Prefer a new topic and consumer group, or deliberately reset offsets after restoration.

  • Ordering: Kafka guarantees ordering only within a partition. When several source partitions are merged, ensure records for the same entity are written to the new topic in their original order.

  • Consumer lag: A smaller partition count reduces consumer parallelism and increases per-partition processing load. Validate throughput and lag before cutover.

  • Replication: Maintain an appropriate replication factor and min.insync.replicas. Monitor UnderReplicatedPartitions, OfflinePartitionsCount, disk usage, request latency, and consumer lag.

  • Security: Grant READ, WRITE, and DESCRIBE on orders-v2 to the relevant clients, CREATE on CLUSTER to topic administrators, and DELETE on orders to principals responsible for cleanup. Grant consumer-group ACLs separately as applicable. Preserve TLS, SASL, quota, and topic-configuration policies.

  • Downstream systems: Update Kafka Connect source/sink topics settings, Kafka Streams topologies, schemas, and Schema Registry subjects. With TopicNameStrategy, a new topic normally requires a new subject name and compatibility decision.

  • KRaft: The restriction is unchanged in KRaft mode; controllers enforce the same topic metadata operations.

Best Practices

  • Prefer a new topic and controlled migration over deleting and recreating a production topic in place.

  • Record the original topic configuration, partition assignments, ACLs, quotas, schemas, and consumer positions before migration.

  • Use idempotent producers and idempotent or transactional downstream writes where appropriate to control duplicates.

  • Keep the source topic until record counts, checksums, key ordering, lag, and business reconciliation all pass.

  • Load-test the smaller partition count; reducing partitions can create hot partitions and remove the parallelism needed for peak throughput.

45

How does adding Kafka partitions affect message ordering by key?

Adding partitions can break end-to-end ordering for a key because Kafka guarantees ordering only within a partition. Existing records remain in their original partitions, while the default producer partitioner may route new records for that key to a different partition after the partition count changes.

Read full answer

Applies toApache Kafka brokers and Java clients 3.x–4.x (KRaft and ZooKeeper-based deployments in 3.x; KRaft-only in 4.x)

How Partition Selection Changes

With the default Java producer partitioner, a non-null key is assigned using the positive Murmur2 hash of its serialized bytes and the topic’s current partition count. Conceptually:

partition = toPositive(murmur2(serializedKeyBytes)) % partitionCount

Increasing the count changes the modulus, so the same key may map to a different partition. Custom partitioners can implement different, potentially stable, mappings.

Situation Result
The key remains in the same partition Ordering for that key continues in that partition
The key maps to another partition Historical and new records for the key are split across partitions
The two partitions are consumed by one group The newer record may be processed first
The key is null There is no key-order guarantee; the default sticky partitioner may distribute such records among partitions

Kafka does not move or rewrite existing records when partition count increases. Once a producer observes the updated metadata, it uses the new count; a producer with stale metadata can temporarily continue sending to the old mapping.

Consumer and Delivery Effects

  • A consumer group must assign the new partition, normally through a rebalance, and may process it concurrently with existing partitions.

  • Consumer offsets are tracked per TopicPartition; offsets from different partitions cannot establish relative ordering.

  • Two partitions can represent different time ranges, so a consumer can observe records for one key as A, C, B.

  • acks=all waits for the current in-sync replicas to acknowledge, but fault-tolerant durability also depends on the replication factor and min.insync.replicas. It never provides cross-partition ordering.

  • Idempotence prevents duplicates caused by retries within a producer session and preserves ordering within a partition when correctly configured. It requires enable.idempotence=true; compatible settings include acks=all, retries>0, and max.in.flight.requests.per.connection no greater than 5.

  • The usual consumer-group pattern of processing before committing offsets is at-least-once. Kafka’s exactly-once processing applies to Kafka consume-transform-produce flows using a transactional producer (transactional.id), enable.auto.commit=false, isolation.level=read_committed, and sendOffsetsToTransaction. It does not automatically cover external side effects.

  • If a key has been split across partitions and that split cannot be prevented, consumers need an application-level sequence number, version, or causal dependency to detect and handle reordering.

Production Operations

  • Plan the rebalance: New partitions change group assignments; eager rebalancing can temporarily move all assignments, while cooperative assignment can reduce unnecessary movement.

  • Check partition health: A new partition starts without historical records and may temporarily have leader-election or ISR convergence delays.

  • Monitor the change: Track under-replicated and offline partitions, producer latency, request rate, consumer lag, and rebalance duration.

  • Expect persistent skew: Hash remapping can redirect a popular key’s new traffic to one new partition indefinitely while its historical data remains in the old partition.

  • Avoid uncontrolled expansion: Draining the old partitions before expanding does not help on its own: after expansion a key can still map to a new partition, so its history stays split. If strict per-key continuity is required, use one of the approaches below instead of a plain partition increase.

  • Remember topic configuration: num.partitions controls the default partition count for newly auto-created topics; it does not increase the count of an existing topic.

Preserving Key Ordering

There is no broker-side operation that automatically merges the histories of a key’s old and new partitions. Practical approaches are:

  • Use stable explicit routing: A custom partitioner can map each key to a fixed physical partition, preserving existing assignments but limiting the parallelism gained from additional partitions.

  • Perform a controlled migration: Pause producers, create a destination topic with a compatible partition count and partitioning scheme, replay records in per-key order, wait for consumers to catch up, cut over, and then resume production.

  • Use application-level ordering: Include an idempotency key and monotonic sequence or version so consumers can deduplicate, buffer, and reject stale records.

  • Serialize affected keys: Route sensitive keys through an ordered processing mechanism instead of allowing independent Kafka partitions to process them concurrently.

46

What is the role of Kafka Connect in Apache Kafka?

Kafka Connect is a framework for running source and sink connectors that move data between Kafka and external systems. Source connectors publish external data into Kafka, while sink connectors consume Kafka records and write them to databases, queues, search indexes, or other services.

Read full answer

Applies toApache Kafka 3.9 and 4.x

Core Role

Kafka Connect standardizes the operational work required to integrate systems with Kafka without building a custom deployment pipeline.

Component Responsibility
Connector Defines the integration logic and its configuration
Source connector Reads from an external system and produces Kafka records
Sink connector Consumes Kafka records and writes to an external system
Worker Hosts connector tasks and manages their lifecycle
Task Executes the actual connector work
REST API Starts, stops, reconfigures, and inspects connectors and tasks

A connector may run as one task or multiple tasks. The tasks.max configuration controls the maximum number of tasks, allowing a sink connector to consume partitions in parallel.

Architecture and Processing

In distributed mode, several Kafka Connect workers form a cluster. They coordinate task ownership through Kafka group-coordination mechanisms and store connector configuration, offsets, and status in internal Kafka topics.

The default internal topic names are:

  • connect-configs: connector and task configuration

  • connect-offsets: source connector offsets

  • connect-status: connector and task status

These names are configurable through the corresponding worker properties. The topics contain operational state, are normally compacted, and should not be manually deleted during normal operation. Their replication factors should be chosen for the available brokers and the required durability.

A source connector reads from an external system and produces records to Kafka. Connector-defined source offsets, such as a database log position, are persisted in Kafka-backed offset storage so that a restarted task can resume from its last committed position. A sink connector consumes Kafka records as an ordinary consumer group (named connect-<connector-name> by default) and commits its offsets to __consumer_offsets.

Converters serialize keys and values between Kafka and the Connect runtime. Connect is separate from Kafka Streams, which is an application library for stateful stream processing. Schema Registry is also separate, although Connect converters can integrate with it.

Delivery and Ordering Semantics

Kafka Connect commonly provides at-least-once processing:

  • A task may complete an external write but fail before committing its Kafka offset.

  • After a restart, the same record can be processed again.

  • Duplicate handling therefore depends on the connector and destination system.

Kafka’s ordering guarantee applies within a partition, not globally across a topic. Concurrent processing can reorder records across partitions, while records within each partition remain ordered.

Exactly-once processing is connector-specific rather than automatic. For source connectors (Kafka 3.3+), the workers must set exactly.once.source.support=enabled, and the connector can then require it with exactly.once.support=required if it supports exactly-once. Sink connectors have no such switch: they achieve exactly-once only if the connector writes idempotently or stores offsets atomically in the destination. An idempotent producer alone does not provide end-to-end exactly-once semantics, and an external database write is not automatically included in a Kafka transaction. Such writes require connector-specific idempotence or transaction integration.

Deployment Modes

connect-standalone.sh runs one worker process and stores offsets locally. It is suitable for development and small, single-process deployments.

connect-distributed.sh runs a scalable worker cluster and stores state in Kafka. Distributed mode is the normal production choice because it provides task redistribution, persistent state, centralized configuration, and REST-based management.

Production Operations

  • Monitoring: Enable JMX metrics, collect worker logs, and alert on failed tasks, repeated restarts, unhealthy internal topics, and consumer-group lag for sink workloads.

  • Failure handling: A failed task is normally restarted and may be reassigned to another worker. Retry limits, backoff policies, dead-letter queues, and recovery procedures are connector-specific.

  • Security: Use TLS and SASL for Kafka connections, apply ACLs to connector and internal topics, and restrict the REST API with network controls, authentication, or a secured reverse proxy.

  • Scaling: Increase tasks.max only when the connector supports parallel processing and the source or destination can handle the resulting concurrency. Scaling workers alone does not create more connector tasks.

  • Upgrades: Canary worker and plugin upgrades, verify connector compatibility, and monitor task reassignment. Keep exports of connector configurations and a recovery plan for internal topic data.

  • Operational pitfalls: Do not delete internal topics during incidents, assume global ordering, expose an unauthenticated REST listener publicly, or describe Connect as providing universal exactly-once behavior.

47

What are source and sink connectors in Kafka Connect?

A source connector reads data from an external system and publishes it to Kafka, while a sink connector consumes Kafka records and writes them to an external system. Kafka Connect runs each connector as one or more tasks managed by Connect workers.

Read full answer

Applies toApache Kafka 4.x (Kafka Connect)

Responsibilities and Data Flow

Connector Data flow Typical example
Source connector External system → Kafka topic PostgreSQL change data capture → Kafka
Sink connector Kafka topic → External system Kafka topic → OpenSearch
MirrorMaker 2, implemented as a source connector Source Kafka cluster → target Kafka cluster Cross-region topic replication

A source connector implementation commonly polls a database, broker, filesystem, or SaaS API. A sink connector invokes connector-specific writes such as inserts, updates, deletes, or service requests. Connect does not automatically make those external writes idempotent; the connector and destination must prevent harmful duplicates or provide reconciliation.

Kafka Streams is different: it is a stream-processing library embedded in an application, whereas Kafka Connect is a worker-based runtime for integrating Kafka with external systems.

Runtime and Delivery Guarantees

Kafka brokers host partitioned, replicated topic logs. Source connectors act as producers, while sink tasks consume assigned topic partitions as part of Kafka consumer groups.

  • Worker: Runs the Connect runtime, connector plugins, and tasks.

  • Connector: Defines the connector class and its configuration.

  • Task: Performs the actual work. tasks.max sets an upper bound rather than guaranteeing that many tasks; sink task count is also constrained by partition assignment.

  • Offset storage: Connect stores source positions and sink offsets in its internal offsets topic, __connect_offsets by default. The name is configurable with offset.storage.topic.

  • Converters: Convert keys and values between Kafka bytes and the connector’s data model. JSON, Avro, and Protobuf converters are available.

  • Schema Registry: Stores schemas and supports compatibility enforcement when used with compatible converters.

Connect normally provides at-least-once delivery. A source can publish a record and fail before its position is committed, causing republication. A sink can complete an external write and fail before committing its Kafka offset, causing the record to be processed again.

Exactly-once operation requires additional support:

  • Configure every worker consistently with exactly.once.source.support=required for sources or exactly.once.sink.support=required for sinks.

  • The connector must implement the corresponding exactly-once contract and configure an idempotent transactional producer with an appropriate unique transactional ID.

  • Kafka transactions make Kafka-side records and offsets atomic. Consumers that must exclude aborted or uncommitted transactional records should use consumer.isolation.level=read_committed.

  • For sinks, a Kafka transaction does not automatically make an external-system write atomic with the offset commit. The connector or destination must also support transactional integration, idempotent writes, or reconciliation.

Kafka preserves ordering only within a partition. Sink tasks may process different partitions concurrently, so global ordering must not be assumed; producers should use a partition key when related records must remain ordered together.

Operational Configuration

  • tasks.max: Increase parallelism only when the connector supports concurrent task execution.

  • consumer.enable.auto.commit: Worker-scoped control for sink-consumer auto-commit behavior; do not use the bare enable.auto.commit name in worker configuration.

  • offset.flush.interval.ms: Controls how frequently the worker flushes connector offsets.

  • errors.tolerance: Determines whether failed records stop processing or are tolerated.

  • errors.deadletterqueue.topic.name: Selects the dead-letter topic when error tolerance permits rejected records to be skipped.

  • schema.registry.url: Points compatible converters to Schema Registry.

  • Credentials: Store database passwords, API tokens, and cloud credentials in an approved secrets mechanism rather than source control.

Production Operations

  • Monitoring: Export JMX metrics and alert on failed tasks, repeated retries, sink consumer-group lag, and dead-letter-topic growth. Check task state through GET /connectors/{name}/status.

  • Data handling: Use replicated topics in production. Apply time or size retention to ordinary event topics; use log compaction for keyed state or Kafka Streams changelog topics.

  • Failure isolation: Configure bounded retries and a dead-letter topic for poison records. Give the dead-letter topic appropriate retention, replication, and access controls.

  • Security: Use TLS and SASL between Connect and Kafka, least-privilege Kafka ACLs, encrypted connector-to-destination traffic, and rotated credentials supplied through a secrets manager.

  • Upgrades: Test connector plugins independently, verify Kafka and plugin compatibility, upgrade distributed workers in a rolling deployment, and redeploy connectors only after the workers are healthy.

  • Recovery: Export connector definitions and plugin configuration through controlled configuration management, back up relevant source systems and schemas, and replicate the internal offsets topic appropriately.

  • KRaft: Apache Kafka 4.x uses KRaft mode, and Connect workers connect through bootstrap.servers rather than ZooKeeper.

48

How does Debezium change data capture work with Kafka?

Debezium extends change data capture by converting database-log changes into Kafka records. Its source connectors usually run in Kafka Connect, while Kafka Connect sinks, Kafka Streams applications, Schema Registry, and ordinary consumers process the resulting event stream.

Read full answer

Applies toDebezium 3.x with Apache Kafka 3.8.x or 4.x; PostgreSQL 13+ for the PostgreSQL-specific flow

CDC Architecture and Flow

  • Database log: PostgreSQL records changes in its write-ahead log. Debezium uses the pgoutput logical-replication plugin and emits only transactions that commit.

  • Publication: The PostgreSQL publication identifies the captured tables. The connector can manage it according to publication.autocreate.mode, or use an existing publication named by publication.name.

  • Replication slot: The logical replication slot, configured with slot.name, preserves required WAL data until Debezium confirms consumption. A disconnected or failed connector can therefore cause WAL accumulation.

  • Snapshot and streaming: The PostgreSQL connector normally takes an initial snapshot and then continues with incremental changes. Inserts, updates, and deletes use op values c, u, and d; snapshot reads use r.

  • Event envelope: A change event contains fields such as before, after, source, op, and ts_ms, with transaction metadata when enabled.

  • Kafka topic: With topic.prefix=inventory and table public.products, the destination is inventory.public.products. Kafka stores each topic as partitioned, replicated logs; the default Debezium key uses the table identity and primary-key columns.

Debezium records source progress in Kafka Connect’s offset store. Downstream Kafka consumers use consumer groups and commit their own offsets independently.

Delivery and Ordering Guarantees

Mode Behavior Requirements
At-least-once A restart can cause an already captured event to be published again Idempotent consumers or deduplication
Exactly-once to Kafka Debezium atomically publishes records and advances source offsets in a Kafka transaction Connector support, a transactional distributed Connect worker, and consumers using isolation.level=read_committed

Exactly-once source behavior is optional and must be enabled with exactly.once.source.support=required. For a highly available Kafka cluster, configure transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2.

Kafka preserves order only within a partition. With the default primary-key partition key, successive changes for the same row normally remain ordered in one partition; no ordering is guaranteed across partitions or tables. Tables without primary keys require an explicit key strategy if row-level ordering matters.

Kafka transactions do not make writes to an external database atomic with Kafka. End-to-end exactly-once applies only when the downstream processing and offset updates participate in the transaction, such as supported Kafka Streams flows; external sinks still require idempotency, transactional CDC, or reconciliation.

Kafka Integration

  • Kafka Connect: Hosts Debezium source tasks and can host sink connectors that write the events to databases, search indexes, or other systems.

  • Kafka Streams: Consumes change topics for filtering, aggregation, enrichment, and materialized views.

  • Schema Registry: Can store Avro or JSON Schema schemas used by compatible converters. Standard JSON, Avro, and JSON Schema converters have different registry behavior, so converter selection must match the deployment.

  • Consumer groups: Partition assignment and consumer offsets are handled by Kafka, not by Debezium.

Production Practices and Failure Modes

  • Kafka topology: Run at least three brokers for a replicated production deployment, pre-create topics with replication factor 3, and normally set min.insync.replicas=2. Use KRaft for metadata management; it is required in Kafka 4.x. Do not depend on broker auto-topic creation.

  • Connect HA: Use multiple Connect workers and monitor /connectors/inventory-source/status. Monitor task failures, restarts, record counts, snapshot progress, and producer errors.

  • CDC lag: Track source lag using Connect connector metrics and the difference between PostgreSQL’s current WAL position and the slot’s confirmed_flush_lsn. kafka-consumer-groups.sh --describe is appropriate for downstream Kafka consumers, not for the database-to-Kafka source lag.

  • Slot health: Monitor active, confirmed_flush_lsn, wal_status, and safe_wal_size in pg_replication_slots. Do not delete slots during routine cleanup. An invalidated slot caused by lost WAL can require a new snapshot.

  • Retention: Retain CDC topics long enough for required replay. Log compaction is useful for latest-state consumers, but it can remove intermediate changes needed for auditing or replay.

  • Schema changes: Test DDL and schema evolution. Breaking changes can stop the connector or make existing events unreadable to consumers, so deploy compatible schemas and coordinated consumer changes.

  • Security and recovery: Use TLS and SASL where appropriate, restrict the Connect REST API, grant the connector only required database privileges, and inject secrets from a secret manager. Protect PostgreSQL with base backups and WAL archiving for point-in-time recovery; Kafka does not replace database backups. Test rolling Kafka and connector upgrades, including offset compatibility and snapshot behavior.

49

What is Schema Registry, and why use it in Kafka?

Schema Registry is a centralized service for storing, validating, and retrieving data schemas used by Kafka producers and consumers. It enables governed schema evolution, preventing incompatible message formats from silently breaking event processing.

Read full answer

Applies toConfluent Platform 8.0.x (Schema Registry 8.0.x) with Apache Kafka 3.8.x or 4.0.x

Core Responsibilities

Schema Registry manages schemas in formats such as Avro, JSON Schema, and Protobuf. Although commonly used with Apache Kafka, it is a separate Confluent project rather than a component built into Kafka brokers.

It provides:

  • Schema storage: Schemas are persisted in the internal _schemas Kafka topic.

  • Schema validation: New versions can be checked against compatibility rules before registration.

  • Schema discovery: Producers and consumers can retrieve schemas through the REST API.

  • Serialization support: Confluent serializers embed a schema ID in each message, allowing consumers to locate the correct schema.

  • Governance: Centralizing schemas gives platform teams one place to enforce contracts, ownership, and lifecycle policies.

The REST API is commonly structured around endpoints such as GET /subjects/{subject}/versions, GET /schemas/ids/{id}, and POST /subjects/{subject}/versions.

How It Works

A typical event-processing flow is:

  1. A producer selects a subject, often using TopicNameStrategy or another naming strategy.

  2. The serializer obtains the schema’s immutable ID from Schema Registry.

  3. The message includes that ID alongside the serialized payload.

  4. A consumer fetches the schema by ID, caches it locally, and deserializes the payload.

Schema Registry does not store every Kafka message. It stores only schema metadata, while Kafka stores the actual records.

Subject organization is important:

  • Subject: A logical grouping of schema versions, commonly one per topic or record type.

  • Version ID: Identifies a specific version within a subject.

  • Schema ID: Identifies the exact schema used to encode a message.

  • Compatibility policy: Determines whether a new subject version is acceptable.

A service should generally use one stable naming strategy. Changing strategies can create separate subjects for the same logical contract and fragment governance.

Compatibility Policies

Compatibility checks occur when a new schema version is registered.

Policy Requirement
NONE No compatibility validation
BACKWARD A consumer using the new schema can read data written with the previous schema
BACKWARD_TRANSITIVE The new schema is backward-compatible with every previous version
FORWARD A consumer using the previous schema can read data written with the new schema
FORWARD_TRANSITIVE The new schema is forward-compatible with every previous version
FULL The new schema satisfies both backward and forward compatibility with the previous version
FULL_TRANSITIVE Both guarantees apply across every previous version

For Avro, BACKWARD is a common starting point because adding a field with a default value can allow newer consumers to read older records. Compatibility can be configured globally and overridden for individual subjects through the /config/{subject} endpoint.

Production Operations

Run at least two Schema Registry instances behind a load balancer and use a replicated _schemas topic. Important server settings include:

  • kafkastore.bootstrap.servers: Kafka brokers used for schema storage.

  • kafkastore.topic: Internal topic name, normally _schemas.

  • kafkastore.topic.replication.factor: Replication factor, normally at least 3 across availability zones.

  • listeners: Schema Registry listener, such as http://0.0.0.0:8081 behind a TLS-terminating proxy.

All instances must use the same backing Kafka cluster and configuration. Monitor:

  • Registry request latency, error rate, and restart count.

  • Compatibility rejection rates.

  • _schemas partition leadership, under-replicated and offline partitions, partition size, and compaction health.

  • Broker disk capacity and request queue depth.

  • Producer and consumer schema-fetch failures.

Back up the _schemas topic, including its subject-level compatibility configuration, to an off-cluster location, and test restoration. Kafka replication protects against broker failure but is not a substitute for backups. During upgrades, canary the new version, verify client compatibility, and monitor registration and retrieval errors.

Security and Failure Modes

Expose Schema Registry only over TLS, require authentication, and authorize access to sensitive subjects. The connection from Schema Registry to Kafka should also use TLS and appropriate ACLs. Never embed credentials directly in schema.registry.url; use the Confluent client’s supported credential mechanism, such as basic.auth.credentials.source=USER_INFO.

Common failure modes include:

  • Registry outage: Producers needing a previously unseen schema may fail. Producers with an already cached schema ID may continue serializing; consumers can continue only if their client has cached the schema metadata required for deserialization.

  • Incompatible registration: The new schema is rejected, protecting existing consumers but potentially blocking deployment.

  • Incorrect subject naming: Equivalent schemas are split across multiple subjects, weakening compatibility enforcement.

  • Schema-ID loss: Losing _schemas data can make existing message IDs unresolvable.

  • False confidence: Schema compatibility checks structure, not business meaning. A schema can be technically compatible while containing invalid values.

Use Schema Registry when independently deployed producers and consumers need a shared, governed contract. For low-risk internal systems with stable formats, it may add operational overhead, but it is especially valuable for event-driven platforms, shared APIs, regulated data, and long-lived consumers.

50

Comparing Avro, Protobuf and JSON Schema for Kafka messages

Avro, Protobuf and JSON Schema provide different trade-offs between schema enforcement, evolution, interoperability, payload size, and operational complexity. Avro has the strongest Kafka-specific ecosystem, Protobuf is efficient for typed contracts, and JSON Schema is flexible for externally governed or dynamic data.

Read full answer

Applies toApache Kafka 4.x and Confluent Schema Registry 7.x when used independently

Format Comparison

Concern Avro Protobuf JSON Schema
Representation Binary, schema-driven Binary, typed messages Textual JSON with an external schema
Schema enforcement Strong field-level contract Strong typed contract Rich validation rules
Kafka integration Mature serializers, converters, and tooling Supported through serializers and Schema Registry Supported through validators and Schema Registry
Typical size Compact binary Usually compact binary Largest because of JSON syntax and repeated field names
Evolution model Reader/writer schema resolution Field numbers and wire compatibility Constraint- and property-based compatibility
Human readability Low Low High
Best fit Event streaming and data pipelines High-throughput typed systems Heterogeneous APIs and dynamic payloads

Avro represents records as named fields and types, with support for unions and defaults. Its reader/writer schema resolution provides explicit compatibility rules.

Protobuf encodes field numbers and wire types instead of field names. Compatibility depends on preserving field numbers and using wire-compatible changes.

JSON Schema describes and validates a JSON document. It is not itself a binary serialization format, and a JSON message can exist without any machine-enforced schema.

Kafka and Schema Registry

Kafka brokers do not understand these schemas. Serialization and validation occur in producers, consumers, Kafka Connect converters, or Kafka Streams applications.

Confluent Schema Registry stores versioned schemas under subjects such as orders-value and orders-key. Registry-aware serializers typically attach a schema ID to the payload; clients retrieve the corresponding schema and cache it locally.

  • Avro: The payload uses Avro binary encoding, while Schema Registry manages the writer schema and its ID.

  • Protobuf: The payload uses Protobuf wire encoding, while Schema Registry manages the schema definition and compatibility checks.

  • JSON Schema: The payload remains JSON, and a validator can enforce the registered schema before transmission or consumption.

Compatibility checking is format-specific. Avro has defined BACKWARD, FORWARD, and FULL reader/writer semantics. Protobuf checking focuses on field-level and wire compatibility. JSON Schema checking depends on the draft and the constraints being changed. A successful registry check does not prove that existing consumers preserve the intended business semantics.

Evolution and Delivery Guarantees

  • Avro evolution: Add a new field with a reader-side default when old readers must continue working. Removing fields, introducing incompatible unions, or making changes that readers cannot resolve requires coordinated reader updates.

  • Protobuf evolution: Never reuse a field number for a different meaning. Reserve removed field numbers and names, and assess any type change for wire and semantic compatibility.

  • JSON Schema evolution: Adding an optional property is normally compatible only if existing schemas permit additional properties. Tightening required, types, ranges, patterns, or size constraints can break existing producers and consumers.

Schema choice does not change Kafka’s delivery guarantees:

  • At-least-once: A consumer processes a record before committing its offset. A failure after processing but before the commit can cause redelivery and duplicates.

  • Exactly-once within Kafka: For consume-transform-produce workflows, use an idempotent transactional producer, keep transactional.id stable per logical producer, submit consumed offsets atomically with output records using send_offsets_to_transaction, and set consumers to isolation.level=read_committed. Kafka Streams can provide the equivalent transactional processing model. Arbitrary external side effects still require an idempotent or transactional design.

  • Ordering: Kafka preserves ordering within a partition, not across a topic. Consumers must also be able to deserialize each format’s schema evolution, while application-level semantic compatibility remains a separate concern.

Production Trade-offs

  • Performance: Protobuf is often attractive for small numeric messages, while Avro is competitive and has stronger Kafka-specific tooling. JSON generally consumes more CPU, bandwidth, and storage. Benchmark representative payloads rather than relying on fixed size ratios.

  • Connect and Streams: Use compatible Avro, Protobuf, or JSON Schema converters consistently. Kafka Streams state stores and changelog topics also require compatible serialization.

  • Security: Protect Schema Registry with TLS and authentication, restrict schema-write permissions, and apply Kafka ACLs to topics. Revalidate untrusted JSON because permissive schemas may allow unexpected properties or oversized values.

  • Monitoring: Track serialization failures, registry latency, compatibility rejections, broker request latency, consumer lag, record size, and transaction abort rates.

  • Retention and recovery: Configure topic replication and retention independently of the schema format. Schema Registry persists schemas in Kafka’s internal _schemas topic; ensure Kafka backup and disaster-recovery procedures cover that metadata and test restoration.

Operational Best Practices

  • Register schemas in CI/CD before deploying producers; avoid automatic registration in production.

  • Exercise old and new schema versions against representative consumers in compatibility tests.

  • Give every domain a deliberate subject naming strategy and ownership model.

  • Set explicit compatibility policies and require review for breaking changes.

  • Cache schemas locally, but design for registry outages, unknown schema IDs, and new schema registration failures.

  • Keep each transactional.id stable across restarts of the same logical producer while ensuring that concurrent producer instances do not share it.

  • During incidents, check schema compatibility and serialization errors before assuming the broker or consumer group caused the failure.

  • Upgrade clients and Schema Registry through rolling tests, retaining previous schema versions until all consumers have migrated.

51

What are schema compatibility modes in Confluent Schema Registry?

Schema Registry compatibility modes decide whether a new schema version may be registered for a subject. The registry compares the candidate schema against previously registered versions and rejects the registration (HTTP 409) if the change would break consumers or producers that still use the older schema. The available modes are BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, plus NONE to disable checking.

Read full answer

Applies toConfluent Schema Registry 7.x with Apache Kafka 3.x (also the model used by Apicurio Registry and Karapace)

How the Check Works

  • Schemas are stored per subject (e.g. orders-value), each subject has one schema type (AVRO, JSON, PROTOBUF) and an ordered list of versions.

  • On POST /subjects/{subject}/versions, Schema Registry runs the compatibility checker for that subject’s type: it resolves the candidate schema against the stored versions according to the subject’s mode.

  • Non-transitive modes compare only against the latest version; transitive modes compare against every previous version.

  • The mode is set per subject (PUT /config/{subject}); the registry-wide default comes from schema-registry.properties (schema.compatibility.level; in Docker images it is set with SCHEMA_REGISTRY_SCHEMA_COMPATIBILITY_LEVEL). The default is BACKWARD.

  • Changing the mode is metadata-only: already-registered versions are not re-validated, so tightening the mode does not retroactively invalidate anything.

The Modes

Mode Compared against Meaning
BACKWARD latest version New schema can read data written with the previous schema (new consumers can read old data)
BACKWARD_TRANSITIVE all versions Same rule, but checked against every historical version
FORWARD latest version Previous schema can read data written with the new schema (old consumers keep working with new producers)
FORWARD_TRANSITIVE all versions Same rule, checked against every historical version
FULL latest version Both backward and forward compatible
FULL_TRANSITIVE all versions Both directions, checked against every historical version
NONE — No checks; any schema is accepted

Choosing a Mode by Schema Type

Change BACKWARD FORWARD Notes
Avro: add field with default Yes Yes Compatible both ways (FULL): new readers use the default, old readers ignore the field
Avro: add field without default No Yes New readers have no value for old data; old readers just ignore the extra field
Avro: remove field Yes Only if the old schema had a default for it Otherwise the old reader cannot reconstruct the value
Avro: add enum symbol Yes Only if the old (reader) schema defines an enum default Old readers fall back to their default for unknown symbols
Avro: rename field No No Use aliases instead
Protobuf: add optional field Yes Yes Never reuse a deleted field number
JSON Schema: add optional property Yes Yes Adding to required breaks backward compatibility

Configuring and Testing with REST

# Current mode for a subject
curl -s http://schema-registry:8081/config/orders-value

# Set the mode for a subject (or PUT /config for the registry-wide default)
curl -s -X PUT -H 'Content-Type: application/json' \
  --data '{"compatibility":"FULL_TRANSITIVE"}' \
  http://schema-registry:8081/config/orders-value

# Dry-run a candidate schema: nothing is stored
curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total\",\"type\":\"double\",\"default\":0.0}]}"}' \
  'http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest?testCompatibility=true'

# Register the schema for real
curl -s -X POST -H 'Content-Type: application/json' \
  --data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total\",\"type\":\"double\",\"default\":0.0}]}"}' \
  http://schema-registry:8081/subjects/orders-value/versions

Java Code Example

import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;

public class CompatAdmin {
    public static void main(String[] args) throws Exception {
        SchemaRegistryClient client =
                new CachedSchemaRegistryClient("http://schema-registry:8081", 100);

        // Tighten the mode before anyone registers the next version
        client.updateCompatibility("orders-value", "FULL_TRANSITIVE");

        // Producers/consumers can opt out of auto-registration so bad schemas fail fast
        System.out.println("Mode set for orders-value");
    }
}

Operational Guidance and Pitfalls

  • Use BACKWARD_TRANSITIVE or FULL_TRANSITIVE for long-lived subjects: it prevents “schema drift”, where each version is only compatible with its immediate predecessor but breaks consumers on version 1. The cost is that the rule tightens as history grows, so plan migrations early.

  • FULL (non-transitive) is a common middle ground: it protects both directions for the current version while staying easier to evolve than the transitive variant.

  • Set modes per subject, not globally: unrelated subjects often need different rules (for example, an internal compacted topic may use NONE).

  • Roll out in order: deploy consumers that can read both versions first, then register the new schema, then switch producers to it.

  • Set auto.register.schemas=false on serializers in production so an incompatible schema fails at startup instead of being silently registered.

  • Monitor: alert on registration responses with HTTP 409, and watch the internal _schemas topic for compaction and replication lag. Run Schema Registry in HA (3+ nodes) with kafkastore.bootstrap.servers pointing at a production cluster, and back up the _schemas topic — it is the source of truth for all schema history.

  • Common traps: Avro fields without defaults, protobuf field-number reuse, JSON Schema required changes, and assuming a mode change re-validates existing versions.

52

How does the Kafka Streams library process event streams?

Kafka Streams is a Java library for building stateful, real-time applications that consume from and produce to Apache Kafka topics. It runs inside an application’s JVM, with multiple instances coordinating through Kafka.

Read full answer

Applies toApache Kafka 4.x (Kafka Streams 4.x)

Architecture and Programming Model

  • Kafka architecture: Topics are partitioned and replicated across brokers. Kafka 4.x uses KRaft consensus rather than ZooKeeper.

  • Topology: An application defines a directed graph of source, processor, sink, and join nodes through the high-level DSL or lower-level Processor API.

  • Streams and tables: A stream represents individual events; a table represents the latest value for each key. The stream-table duality lets the same data be processed using either abstraction.

  • Partitions and tasks: Partitions are divided into independent stream tasks. Each task processes records sequentially, while multiple task threads provide parallelism.

  • Consumer groups: Instances sharing an application.id form a consumer group and divide source partitions. Additional instances increase parallelism only when enough partitions exist.

  • State stores: Joins and aggregations use local state, commonly RocksDB. Kafka changelog topics provide durable recovery data.

  • Repartitioning: Keys must reach the same partition for key-based joins and aggregations. Kafka Streams creates repartition topics when the topology requires them.

  • Schema Registry: Kafka Streams does not require Schema Registry. Schema-aware Avro or Protobuf serializers and deserializers can be configured independently, with compatibility enforced by the registry.

State, Ordering, and Delivery Guarantees

  • Ordering: Kafka preserves order within a partition, not globally across a topic. Ordered processing therefore requires appropriate partitioning.

  • At-least-once processing: With processing.guarantee=at_least_once, failures can cause records to be processed again. A crash after producing output but before committing the consumed offset can create duplicate output.

  • Exactly-once processing: In Kafka 4.x, processing.guarantee=exactly_once_v2 is the default. Kafka Streams transactionally commits output records, state-store changelog updates, and consumed offsets.

  • Transactional requirements: The broker must support transactions. Consumers of transactional source or output topics should use isolation.level=read_committed to filter aborted records.

  • Scope: Exactly-once applies to Kafka Streams processing, including Kafka-backed state and Kafka output. External side effects require idempotency or external transaction coordination.

  • Commit interval: commit.interval.ms trades transaction overhead against replayed work after failure. Smaller intervals commit more frequently but increase overhead.

Production Operations and Failure Modes

  • State recovery: RocksDB files are local working state, not durable backups. After restart, Kafka Streams restores its latest state from changelog topics and replays subsequent records.

  • Replication: Set the internal-topic replication.factor=3 where appropriate, and verify broker settings such as min.insync.replicas and replication policies.

  • Scaling: Increase source partitions before adding instances. Too few partitions leave instances idle, while a hot partition cannot be subdivided further.

  • Rebalancing: Restarts and membership changes trigger consumer-group rebalancing, partition revocation or reassignment among members, and state restoration. Monitor rebalance duration and restoration cost.

  • Backpressure: Processing lag grows when input exceeds throughput. Tune max.poll.records, task parallelism, and processing performance rather than relying only on lag alerts.

  • Monitoring: Track source lag, processed-record rate, processing latency, commit failures, rebalances, exceptions, changelog growth, and state-store disk usage through JMX or the metrics platform.

  • Upgrades: Follow broker and client compatibility requirements. Test changelog restoration, topology changes, and rolling restarts in staging before production.

  • Backups: Replicated Kafka clusters protect changelog data, but local RocksDB directories are not backups. Back up internal-topic configuration, application configuration, and ACLs.

  • Poison records: Configure deserialization and processing exception handling, route irrecoverable records to a dead-letter topic, and alert on skipped or failed records.

Kafka Streams Compared with Other Kafka Tools

Tool Deployment model Stateful processing Typical use
Kafka Streams Embedded in a Java application Local stores with Kafka changelogs Business logic, aggregations, joins, transformations
Kafka Connect Connector workers Connector-specific Moving data between Kafka and external systems
Apache Flink Dedicated distributed runtime Distributed managed state and checkpoints Large-scale processing with cluster-level control

Kafka Streams fits when processing logic belongs with a Java service and Kafka is already the system of record. Flink is often preferred for very large state, dedicated cluster scheduling, and independent checkpointing control.

Security and Engineering Trade-offs

  • Transport security: Use TLS, commonly with security.protocol=SASL_SSL, and configure the broker truststore. Select a supported mechanism such as SCRAM-SHA-512.

  • Credentials: Supply sasl.jaas.config through secret management rather than committing credentials to configuration files.

  • Authorization: Grant principals only required topic, group, internal-topic, and transactional-ID permissions.

  • Transactions: Ensure transaction.timeout.ms does not exceed the broker’s transaction.max.timeout.ms, and monitor transaction timeouts.

  • Language and operations: Kafka Streams has a native Java API, simplifying deployment but making JVM sizing, thread configuration, and local disk capacity application responsibilities.

  • Failure isolation: A continuously failing topology can block its partitions. Use supervised deployments, bounded retries, dead-letter handling, and alerts distinguishing broker failures from deterministic data errors.

53

What is the difference between KStream and KTable in Kafka Streams?

KStream represents an ordered event stream, while KTable represents the latest value for each key. KStream preserves every record independently; KTable materializes state so newer values replace older ones.

Read full answer

Applies toApache Kafka 4.x (Kafka Streams 4.x)

Core Distinction

Aspect KStream KTable
Data model Sequence of key-value events Current key-value state
Duplicate keys Retained as separate records Collapsed into the latest value
Historical data Retained in the source topic according to its retention policy Not retained in materialized state; the changelog contains update records until compaction removes obsolete versions
Typical use case Transactions, clicks, measurements, logs Profiles, account balances, inventory levels, device state
Internal topics May require repartition topics for stateful aggregations or joins State-store changelog, normally compacted
Update behavior Every record is an event Each update replaces the previous value for that key

For example, sensor readings such as temperature=21, temperature=22, and temperature=23 are three KStream events. The corresponding KTable contains only the latest state: temperature=23.

How They Work Internally

  • KStream: Each input record is processed according to its partition and offset. A stream does not automatically merge records with the same key; that aggregation must be defined explicitly.

  • KTable: Kafka Streams maintains a state store indexed by key. A new record updates the stored value. When another instance needs to restore the table, it reads the table’s changelog topic.

  • Changelog compaction: KTable changelog topics use log compaction to retain the most recent non-null value for each key. Intermediate updates may remain before compaction runs, so storage can temporarily exceed the logical table size.

  • Tombstones: A null value represents deletion in materialized KTable state and produces a tombstone in the changelog. Compaction does not immediately remove a tombstone; it is retained for the topic’s delete.retention.ms and removed afterward.

KTable is therefore an abstraction over changing state, not simply a compacted Kafka topic. Kafka Streams maintains its changelog for restoration and recovery; downstream applications should normally consume application output topics rather than depend on internal changelog topics.

Aggregation and Joins

After grouping records by key, a KStream can be aggregated into a KTable whose result is backed by a state store.

Joins differ depending on the joined abstraction:

Join Operational behavior
KStream–KTable Each stream record is matched against the current table value
KTable–KTable Emits when either side’s materialized value changes
KStream–KStream Matches records from the two streams within the join window; repartitioning may be required to co-partition keys

For joins involving stateful KTables, Kafka Streams uses the table’s materialized state and maintains its changelog. Stream–stream joins and certain stream aggregations may introduce repartition topics to ensure records with the same key reach the same task.

Delivery and Ordering Guarantees

  • Ordering: KStream records are ordered within a partition. A KTable is logically a latest-value view of records processed in that order.

  • At-least-once: Processing can repeat after failures, so downstream consumers should be idempotent when necessary.

  • Exactly-once: Kafka Streams supports processing.guarantee=exactly_once_v2 with a unique, stable application.id; Kafka Streams manages transaction IDs internally. Input offsets, state-store changelog writes, and output records are then committed atomically to Kafka, while local state stores are restored consistently after failures.

  • Limit: Exactly-once does not automatically make an external database or HTTP side effect atomic with Kafka. Such integrations still require idempotency, transactional outbox patterns, or equivalent coordination.

Production Considerations

  • Monitor: Track consumer lag, repartition-topic throughput, restore latency, state-store size, rebalance frequency, and processing latency.

  • Capacity plan: Changelog topics and restored state require disk capacity. Compaction reduces retained history but does not guarantee immediate disk reclamation.

  • Recovery: Configure standby replicas for important state stores so another instance can restore them faster and reduce recovery time.

  • Schema compatibility: Use compatible schemas, typically with Schema Registry, before evolving KStream or KTable payloads.

  • Choose deliberately: Use KStream for independent facts and event processing; use KTable when consumers care about current state rather than every transition.

54

How do Kafka Streams state stores enable stateful processing?

State stores let Kafka Streams maintain task-local state while processing records, enabling keyed aggregations, joins, windowed calculations, and deduplication without holding the entire dataset in the application. Persistent, logging-enabled stores normally publish state changes to a Kafka changelog so state can be restored after restarts or task reassignment.

Read full answer

Applies toApache Kafka 4.x (Kafka Streams 4.x)

How It Works

A stream task owns one or more input-topic partitions and the corresponding partitions of the state stores it uses:

  1. A record is read from an input topic.

  2. The processor reads or updates state through its store.

  3. The updated state is written to the local store and, for a persistent logging-enabled store, to its changelog topic.

  4. Output records are sent to downstream topics.

  5. Consumed offsets are committed according to the configured processing guarantee.

State is generally partitioned by record key, so all records for a key are handled by the same task and can access the same state partition. Store names must be unique within the topology.

When num.standby.replicas is greater than zero, a standby task maintains a copy of an active task’s state by consuming its changelog. If the active task fails, the standby can be promoted with local state already close to current, reducing restoration work.

Store Types and Changelogs

  • RocksDB: The default persistent key-value store. It keeps state on local disk and is appropriate when state should not all reside in the Java heap, subject to adequate disk and I/O capacity.

  • In-memory stores: Faster for small or temporary state. Their contents disappear with the process unless logging and recovery are enabled.

  • Window stores: Retain timestamped values according to window boundaries and retention settings.

  • Session stores: Group records into sessions using inactivity-gap and grace-period rules.

A changelog topic is typically named:

<application.id>-<store-name>-changelog

Changelog topics normally use log compaction so that the latest value for each key can be restored. A null-value tombstone removes the key when the changelog record is processed; compaction later removes older non-null records and eventually removes the tombstone itself according to delete-retention settings.

Changelog retention is independent of window-store retention. Operators must retain enough changelog history for the required recovery horizon; if the necessary records expire, Kafka Streams may not be able to rebuild the complete local state.

Recovery Architecture

In Kafka 4.x, broker clusters use KRaft, and changelog durability depends on normal Kafka topic replication. The local RocksDB directory is therefore not a complete disaster-recovery backup. If local state is lost, Kafka Streams can restore it by replaying the changelog.

Production recovery plans should:

  • Set statestore.changelog.topic.replication.factor appropriately for the broker count.

  • Monitor replication, under-replicated partitions, disk capacity, and standby-task lag.

  • Keep state.dir on reliable storage with sufficient capacity and I/O performance.

  • Protect internal topics with appropriate ACLs.

  • Use a replicated Kafka cluster, MirrorMaker, or an equivalent backup strategy for cluster-level recovery.

Delivery Semantics and Ordering

Concern Kafka Streams behavior
Default processing guarantee At-least-once processing; a crash may cause a record to be processed again
processing.guarantee=exactly_once_v2 Kafka output records, changelog writes and consumed offsets are committed atomically (exactly_once_v2 is the only exactly-once value in Kafka 4.x)
Ordering Records are processed in order within each input partition
Multiple partitions No global ordering is provided

Exactly-once semantics apply to Kafka-to-Kafka processing. Local and changelog state are made consistent through Kafka Streams’ checkpoint and restoration machinery; the changelog should not be treated as an exactly-once side effect for an external system. Output consumers should use isolation.level=read_committed to avoid observing aborted transactions.

Exactly-once operation also requires suitable broker transaction-state-log settings, including transaction.state.log.replication.factor and transaction.state.log.min.isr, plus the necessary transactional producer and ACL permissions. Writes to external databases or HTTP systems require separate idempotency or transaction mechanisms.

Production Practices and Failure Modes

  • Monitor task health: Track input lag, standby lag, processing latency, commit latency, RocksDB metrics, and state-directory disk usage.

  • Control resource usage: Excessive cache pressure, compaction backlog, or slow disks can cause task lag and restarts.

  • Plan rebalances: Adding instances or changing stream-thread and partition assignments can trigger state movement and restoration.

  • Handle tombstones deliberately: Configure serializers and application logic so null values mean deletion rather than accidental data loss.

  • Secure the platform: Use TLS and SASL, restrict access to application and internal topics, and protect sensitive state at rest.

  • Test upgrades and recovery: Verify rolling upgrades, restart restoration, partition reassignment, and full state reconstruction rather than assuming local RocksDB files are sufficient.

55

How does Kafka Streams aggregate records within event-time windows?

Event-time windowed aggregation is Kafka Streams’ mechanism for grouping records by key and timestamp-defined windows while maintaining one aggregate per key-window pair. It uses local state stores for active state and changelog topics to recover or restore that state after failures and rebalances.

Read full answer

Applies toApache Kafka Streams 4.x

Core APIs and Window Types

A typical aggregation flow uses groupByKey(), followed by windowedBy(...), and then an aggregation operation such as aggregate(...), reduce(...), or count(). The result is a KTable<Windowed<K>, VR>, where VR is the aggregate’s value type. For example, a five-minute tumbling window can maintain the total order value for each customer in each interval.

Window type Behaviour Relevant API
Tumbling Fixed, non-overlapping intervals aligned to the epoch TimeWindows.ofSizeWithNoGrace(...) or TimeWindows.ofSizeAndGrace(...)
Hopping Fixed-size windows that overlap by a configured advance interval TimeWindows.ofSizeWithNoGrace(...).advanceBy(...) or TimeWindows.ofSizeAndGrace(...).advanceBy(...)
Session Per-key activity windows separated by an inactivity gap SessionWindows.withInactivityGapAndGrace(...)

A grace period extends a window after its normal end so that late records can still update the aggregate. Increasing grace improves late-data tolerance, but retains state longer and delays the final result.

Runtime Lifecycle

  1. A stream task consumes a record and uses the input record’s timestamp—not processing time—to assign it to the applicable window or windows.

  2. The task looks up the previous aggregate for that key and window in its local state store.

  3. If no aggregate exists, Kafka Streams invokes the initializer. It then applies the aggregator to the record and the previous aggregate.

  4. The updated aggregate is written to the local store and, for materialized state, recorded in a changelog topic.

  5. The logical KTable exposes the update for downstream processing or materialization.

Kafka Streams derives stream time from observed record timestamps while accounting for out-of-orderness. A record is late and dropped when stream time has already passed the end of its window plus the grace period. Session-window grace begins after the per-key inactivity gap ends.

Without suppression, downstream consumers can receive several updates for one window. Suppressed.untilWindowCloses(...) can defer time-windowed results until the window closes, reducing intermediate output.

Delivery Semantics and Guarantees

  • Default processing: Kafka Streams normally provides at-least-once processing. After a failure, a record may be processed again, so aggregators and side effects should be idempotent where possible.

  • Exactly-once processing: Set processing.guarantee=exactly_once_v2 to atomically coordinate consumed offsets, Streams state changes, and Kafka output records. Use an application.id that uniquely identifies the application’s state and a stable state.dir. The broker should also use appropriate transaction.state.log.replication.factor and transaction.state.log.min.isr settings for transaction-state durability.

  • External side effects: Exactly-once processing does not automatically make non-transactional effects, such as an HTTP request or SQL update, exactly once.

  • Ordering: Kafka preserves record order within a partition. Records for one key must therefore reach the same partition when their order matters.

Operational Behaviour

  • Hot keys: A highly active key can skew one partition and one state-store partition, limiting scalability.

  • State growth: Long windows, large grace periods, and high-cardinality keys increase RocksDB disk use, cache pressure, and restore time.

  • Recovery: After a failure or rebalance, restoring a large changelog can delay processing while the affected task becomes active.

  • Retention: Changelog retention must protect the state needed for recovery. Input retention is a separate concern for replaying source data, and changelog compaction does not replace correct retention and state-lifecycle planning.

  • Durability: Use replicated topics, an appropriate min.insync.replicas, and producer durability settings matched to the required loss tolerance.

Production Guidance

  • Monitor consumer lag, processing latency, rebalance duration, state-store size, local disk pressure, and restore time.

  • Size grace periods from measured lateness rather than choosing an unnecessarily large default.

  • Test failure recovery, rebalances, broker replacement, and rolling upgrades with production-like topic and key distributions.

  • Verify exactly-once behaviour end to end when it is required; transaction guarantees apply only to Kafka-backed inputs, outputs, and Streams state.

56

How does ksqlDB process streaming data?

ksqlDB is a streaming database that executes SQL continuously over Apache Kafka records. It transforms, joins, filters, and aggregates events, maintains state where necessary, and publishes results back to Kafka.

Read full answer

Applies toApache Kafka 3.9.x with ksqlDB 0.29.x

Architecture and Data Model

Kafka brokers store partitioned, replicated topic logs; in KRaft mode, a controller quorum manages cluster metadata. ksqlDB remains a query service and Kafka client, not a replacement for those brokers or controllers.

  • Stream: An append-only sequence of events; each row is an independent fact, and a key is optional.

  • Table: A mutable view holding the latest value per key, built from a changelog or compacted topic or by an aggregation.

  • Persistent query: CREATE STREAM ... AS SELECT or CREATE TABLE ... AS SELECT. It runs continuously on the server and writes its results to a Kafka topic.

  • Push query: SELECT ... EMIT CHANGES. It streams new results to the client as events arrive until the client stops it; nothing is written to Kafka.

  • Pull query: SELECT without EMIT CHANGES. It returns the current state once and ends, like a database query, usually as a key lookup against a materialized table.

ksqlDB uses the Kafka Streams runtime internally. Each persistent query is a Kafka Streams application with its own application ID and consumer coordination; source partitions and offsets are managed through Kafka. Stateful queries keep local state, normally in a RocksDB-backed state store, and use compacted changelog topics to restore state after failures.

Push and Pull Queries

Property Pull query Push query
Execution Runs once and terminates Runs until the client cancels it
Output Current result returned to the client A continuous stream of changes sent to the client
State Reads the state store of a table materialized by a persistent query (or scans a stream’s topic) Subscribes to new events; the state itself is maintained by persistent queries
Typical use Point lookups, such as a customer’s current balance Live dashboards and notifications
Consistency Current per-key result; no global cross-partition snapshot Incremental updates as new records arrive

Pull queries are cheapest as key lookups on a materialized table. A pull query that scans a whole table or stream reads every partition and is expensive. Neither form provides a globally consistent snapshot across partitions.

Delivery, Ordering, and Exactly-Once Semantics

  • Kafka preserves record order only within a partition. ksqlDB preserves that order until an operation such as a repartitioning join changes the partitioning.

  • Grouped aggregations emit updates by grouping key; there is no global ordering across output partitions.

  • By default ksqlDB queries are at-least-once, so a failure can publish duplicate output records.

  • For exactly-once processing inside Kafka, set ksql.streams.processing.guarantee=exactly_once_v2 in the server configuration; ksqlDB then uses Kafka Streams transactions for its output, state and consumed offsets. The brokers need a replicated transaction-state log.

  • Downstream consumers should use isolation.level=read_committed so they do not read aborted output. Data that Kafka Connect brings in or writes out is exactly-once only if those connectors support it.

Exactly-once applies to the transactional processing chain, not automatically to arbitrary external side effects performed by a connector or application.

Kafka Connect and Schema Registry

Kafka Connect supplies source and sink connectors for databases, logs, files, SaaS systems, and other external systems. ksqlDB can use an embedded Connect deployment or a separately operated distributed cluster. Connector lag, task failures, retries, and transactional support must be monitored.

Schema Registry stores and validates schemas used by JSON_SR and AVRO_SR. The configured compatibility policy, such as BACKWARD, BACKWARD_TRANSITIVE, or FULL, controls which schema changes are accepted. Unregistered or incompatible records can cause deserialization failures.

Production Operations, Failures, and Security

  • Monitoring: Track query status, processing lag, state-store size, changelog lag, broker disk and request latency, Connect task lag, and schema errors.

  • Capacity: Increase partitions for independent producer throughput, and size state stores, changelog retention, heap, and RocksDB resources for the working set.

  • Retention and compaction: Time or size retention bounds log growth; compaction retains the latest value per key for compacted topics. Compaction is not a backup mechanism, and retention still limits historical replay.

  • Recovery: Protect Kafka and Schema Registry with appropriate replication and backups. Push-query state can normally be rebuilt by restoring the source and changelog data, subject to retention.

  • Upgrades: Test schema, connector, and query compatibility, then upgrade ksqlDB instances in a controlled rolling sequence while verifying state-store recovery.

  • Security: Use TLS or mTLS, SASL authentication, Kafka ACLs, authenticated Schema Registry access, and encrypted ksqlDB listeners. Avoid placing credentials directly in shell history or process arguments.

  • Failure modes: Do not assume global ordering, globally consistent pull-query snapshots, or connector-wide exactly-once delivery unless the required transactional configuration is present.

58

How do you secure Apache Kafka with TLS, SASL, and ACLs?

Secure Apache Kafka by combining TLS for transport encryption, SASL for authentication, and ACLs for topic-level authorization. These controls should be applied to client listeners, inter-broker traffic, and KRaft controller traffic.

Read full answer

Applies toApache Kafka 4.x in KRaft mode

Security Model

  • TLS protects confidentiality and integrity, but does not normally identify clients by itself.

  • SASL authenticates clients and brokers. Prefer SCRAM-SHA-512; use PLAIN only with TLS and where stronger mechanisms are unavailable.

  • ACLs authorize authenticated principals to perform operations on topics, consumer groups, transactional IDs, and cluster resources.

  • Use SASL_SSL rather than SASL_PLAINTEXT. Optional mutual TLS adds certificate-based client authentication, but it does not replace ACLs.

Server Configuration

A minimal combined broker/controller configuration looks like this:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1.example.com:9094

listeners=SASL_SSL://0.0.0.0:9093,CONTROLLER://0.0.0.0:9094
advertised.listeners=SASL_SSL://kafka1.example.com:9093
inter.broker.listener.name=SASL_SSL
controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:SSL,SASL_SSL:SASL_SSL
listener.name.controller.ssl.client.auth=required

ssl.keystore.location=/etc/kafka/secrets/kafka.p12
ssl.keystore.password=CHANGE_ME
ssl.key.password=CHANGE_ME
ssl.truststore.location=/etc/kafka/secrets/truststore.p12
ssl.truststore.password=CHANGE_ME

sasl.enabled.mechanisms=SCRAM-SHA-512
sasl.mechanism.inter.broker.protocol=SCRAM-SHA-512
listener.name.sasl_ssl.scram-sha-512.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="broker" password="CHANGE_ME";

authorizer.class.name=org.apache.kafka.metadata.authorizer.StandardAuthorizer
allow.everyone.if.no.acl.found=false
super.users=User:kafka-admin;User:broker;User:CN=kafka1.example.com

The broker certificate must contain kafka1.example.com in its SAN. The controller listener uses mutual TLS here, so the certificate’s principal (User:CN=kafka1.example.com) is listed in super.users along with the inter-broker SCRAM user. On brokers, the JAAS setting must carry the listener and mechanism prefix, as shown. In production, use separate controller and broker nodes, a three- or five-controller quorum, and inject passwords from a secret manager rather than storing them directly in server.properties.

SASL Users and ACLs

SCRAM credentials are stored as salted, iterated hashes in Kafka metadata. Create the inter-broker and application users with kafka-configs.sh:

bin/kafka-configs.sh \
  --bootstrap-server kafka1.example.com:9093 \
  --command-config admin.properties \
  --alter \
  --entity-type users \
  --entity-name app-writer \
  --add-config 'SCRAM-SHA-512=[password=CHANGE_ME,iterations=8192]'

Apply least-privilege ACLs for a producer and consumer:

bin/kafka-acls.sh --bootstrap-server kafka1.example.com:9093 \
  --command-config admin.properties --add \
  --allow-principal User:app-writer \
  --operation Write --operation Describe \
  --topic payments --resource-pattern-type literal

bin/kafka-acls.sh --bootstrap-server kafka1.example.com:9093 \
  --command-config admin.properties --add \
  --allow-principal User:app-reader \
  --operation Read --operation Describe \
  --topic payments --resource-pattern-type literal

bin/kafka-acls.sh --bootstrap-server kafka1.example.com:9093 \
  --command-config admin.properties --add \
  --allow-principal User:app-reader \
  --operation Read --operation Describe \
  --group payments-consumers --resource-pattern-type literal

Since Kafka 3.0, idempotent producers need only Write on the topic; transactional producers also need Write and Describe on their transactional.id. Use a distinct service account for each workload, and never make application accounts superusers.

Code Example

A confluent-kafka Python client uses the same transport controls:

from confluent_kafka import Producer

errors = []

def delivered(err, _message):
    if err is not None:
        errors.append(str(err))

producer = Producer({
    "bootstrap.servers": "kafka1.example.com:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "app-writer",
    "sasl.password": "CHANGE_ME",
    "ssl.ca.location": "/etc/kafka/ca.crt",
    "enable.idempotence": True,
    "acks": "all",
})

producer.produce(
    "payments",
    key=b"order-42",
    value=b"created",
    on_delivery=delivered,
)

remaining = producer.flush(10)
if remaining:
    raise TimeoutError("Kafka delivery did not complete")
if errors:
    raise RuntimeError(errors[0])

Failure Modes and Operations

  • TLS handshake failure: check certificate expiry, CA trust, SAN/hostname mismatch, protocol negotiation, and advertised.listeners.

  • SASL authentication failure: verify that client and broker mechanisms match and that the SCRAM username exists. Clients may receive only a generic authentication error, so inspect broker logs.

  • Authorization failure: inspect ACLs with kafka-acls.sh --describe; DENY ACLs take precedence over ALLOW ACLs.

  • Rotation: distribute a new CA before rotating leaf certificates, then perform a rolling restart. Rotate SCRAM credentials carefully because clients and servers must remain synchronized.

  • Monitoring: alert on handshake failures, authentication failures, authorization denials, certificate expiry, under-replicated partitions, and consumer lag.

  • Disaster recovery: back up server.properties, JAAS configuration, certificates, and secret references. ACL and SCRAM user metadata is replicated as KRaft metadata, not stored in ordinary topic data.

Bootstrap requires special care: the inter-broker SCRAM user must exist before brokers can authenticate to each other. In KRaft, create it when formatting storage, for example kafka-storage.sh format --cluster-id <id> --config server.properties --add-scram 'SCRAM-SHA-512=[name=broker,password=CHANGE_ME]' (Kafka 3.5+), then create the remaining users with kafka-configs.sh. No production listener should remain on PLAINTEXT.

59

How do you monitor an Apache Kafka cluster?

Monitor Apache Kafka with layered observability: broker and controller metrics, client telemetry, consumer-lag checks, logs, and infrastructure metrics. In KRaft mode, verify both data-plane health and controller-quorum health.

Read full answer

Applies toApache Kafka 4.x in KRaft mode, Prometheus 3.x, and JMX Exporter 1.x

Monitoring Model

Kafka exposes operational data through JMX. A typical production setup exports broker metrics with JMX Exporter, scrapes them with Prometheus, and displays them in Grafana or another observability platform. Client applications should export producer and consumer metrics separately because broker metrics cannot describe every application-level failure.

Monitor these layers:

  • Data plane: partitions, leaders, in-sync replicas, request queues, disk throughput, and network utilization.

  • Control plane: KRaft controllers, quorum health, metadata changes, and controller elections.

  • Clients: producer errors and retries, consumer rebalances, processed-record rates, and committed offsets.

  • Infrastructure: CPU, memory, garbage collection, disk capacity, I/O latency, and network errors.

  • Business signals: end-to-end message age, missing expected partitions, and downstream processing delays.

Metrics and Alerts

Area Important metrics Operational meaning
Replication UnderReplicatedPartitions, OfflinePartitionsCount, IsrShrinksPerSec Replication is behind or a partition has no available leader.
Partition state LeaderCount, PartitionCount, leader-election rate Detects abnormal imbalance, elections, or topology changes.
Broker health ActiveControllerCount, RequestHandlerAvgIdlePercent, request queue and latency metrics Identifies an overloaded broker or unhealthy controller.
Producer Error rate, retry rate, record queue time, request latency Detects rejected writes, unavailable brokers, or slow acknowledgements.
Consumer Consumed-record rate, fetch latency, rebalance count, poll idle time Detects stalled consumers and unstable group membership.
Consumer lag Committed offset versus log-end offset Estimates unprocessed records; also track lag age because record count alone can be misleading.
Infrastructure Disk free space and inodes, I/O latency, CPU, GC pauses, network errors Finds Kafka failures caused by resource exhaustion.

Alert on OfflinePartitionsCount > 0, sustained under-replicated partitions, an unavailable controller, broker loss, exhausted disks, excessive request queues, and consumer lag exceeding business-defined time limits. Prefer SLO-based thresholds over fixed record counts because traffic and partition sizes vary.

CLI Checks

Inspect partition assignments, leaders, replicas, and ISR state:

/opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --describe \
  --topic orders

Inspect a consumer group’s committed offsets, log-end offsets, lag, and assigned members:

/opt/kafka/bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --describe \
  --group order-processors

Verify the KRaft metadata quorum:

/opt/kafka/bin/kafka-metadata-quorum.sh \
  --bootstrap-server kafka-1.example.com:9092 \
  --status

A healthy quorum should report the expected voters and one active controller. CLI commands are useful for diagnosis, but continuous alerting should come from metrics rather than polling these commands.

Code Example

Install JMX Exporter and expose broker metrics on port 7071. Add this to the broker service environment before starting Kafka:

export KAFKA_OPTS="-javaagent:/opt/jmx-exporter/jmx_prometheus_javaagent.jar=7071:/opt/jmx-exporter/kafka.yml"

A basic /opt/jmx-exporter/kafka.yml is:

rules:
  - pattern: 'kafka.server<type=(.+), name=(.+), topic=(.+)><>Value'
    name: kafka_server_$1_$2
    type: GAUGE
    labels:
      topic: '$3'

  - pattern: 'kafka.server<type=(.+), name=(.+), topic=(.+)><>Count'
    name: kafka_server_$1_$2_total
    type: COUNTER
    labels:
      topic: '$3'

Configure Prometheus to scrape every broker:

scrape_configs:
  - job_name: kafka
    scrape_interval: 15s
    static_configs:
      - targets:
          - kafka-1.example.com:7071
          - kafka-2.example.com:7071
          - kafka-3.example.com:7071

Keep port 7071 on a private management network or place it behind an authenticated TLS reverse proxy. Do not expose raw JMX or an unsecured exporter endpoint publicly.

Delivery Guarantees and Failure Modes

  • Ordering exists only within a partition, not across an entire topic.

  • At-least-once processing requires committing a consumer offset after successful processing. A crash between processing and commit can replay records.

  • At-most-once processing commits before processing and can lose work after a crash.

  • Exactly-once Kafka processing requires producer transactions, a stable transactional.id, and isolation.level=read_committed. Applications must include the consumed offsets in the same transaction; Kafka cannot make arbitrary external side effects exactly-once.

  • For durable writes, producers generally use acks=all, idempotence, and a replicated topic with an appropriate min.insync.replicas.

Typical incidents include disk exhaustion preventing replicas from catching up, an ISR shrinking after a network partition, repeated leader elections, controller-quorum instability, consumer rebalance storms, and long producer retries masking broker unavailability.

Production Practices

  • Monitor per-broker and per-topic cardinality carefully; very large topic counts can increase Prometheus cost.

  • Keep dashboards for cluster capacity, replication, request latency, producer health, consumer lag, and KRaft health.

  • Correlate metrics with broker logs and client traces during incidents.

  • Test alerts by stopping a broker, filling a disk, introducing consumer delay, and simulating controller loss in staging.

  • Upgrade brokers in a controlled rolling sequence and verify replication, quorum status, and consumer groups after each step.

  • Use TLS and SASL for client, administrative, and monitoring access; store credentials in a secrets manager and apply least privilege.

60

Which Apache Kafka metrics matter most in production?

The metrics that matter most in production are broker health, replication status, producer and consumer behaviour, consumer lag, storage pressure, and component error rates. The first alerts should fire on offline or under-replicated partitions, growing consumer lag, rising request latency, and sustained error or retry rates.

Read full answer

Applies toApache Kafka 4.x (KRaft mode)

Essential Broker and Replication Metrics

Metric What it indicates Why it matters
Under-replicated partitions In-sync replicas below the configured replication factor Reduced fault tolerance; possible data loss if another replica fails
Offline partitions Partitions with no elected leader Producers and consumers cannot use them
Leader election rate How often leadership changes Frequent elections signal broker instability or network disruption
ISR shrink/expand rate Changes in the in-sync replica set Repeated shrinking means replicas cannot keep up
Controller status Whether a KRaft controller quorum is active Losing quorum blocks metadata changes and recovery
Request latency Time brokers spend on requests Rising percentiles point to overload, slow disks or GC pauses
Broker CPU, memory, GC, disk Resource pressure on brokers Saturation raises latency and triggers leader movement
Network throughput and errors Inter-broker and client traffic Reveals saturation and retransmissions

A healthy cluster has zero offline partitions and no sustained under-replicated partitions; brief increases during rebalancing or replica replacement are normal, while a growing count is an incident.

Producer Metrics

Watch these at broker and client level:

  • Produce error rate: authorization failures, invalid messages, size violations, unavailable partitions.

  • Retry rate: sustained retries point to unavailable brokers or metadata problems.

  • Record queue time: how long records wait before transmission; high values mean a CPU-bound producer or slow metadata lookup.

  • Batch size and compression ratio: tiny batches hurt throughput, oversized batches raise latency and can trigger MessageSizeTooLargeException.

  • Produce request latency: alert against SLOs, not a fixed threshold, because hardware and message sizes vary.

  • Idempotence and transaction errors: applications needing duplicate prevention must watch sequence-number errors and aborted transactions.

With acks=all, replication completes only when the required in-sync replicas acknowledge the record, so producer success alone can hide a dangerously small ISR.

Consumer and Consumer-Group Metrics

Consumer lag is the main capacity and freshness metric, computed as log end offset - committed consumer offset. Always break it down per partition: a large aggregate can hide one hot partition.

  • Poll rate: near zero can mean a stalled consumer.

  • Records consumed per second: confirms processing progress.

  • Rebalance count and duration: frequent rebalances cause stop-the-world pauses and duplicate work.

  • Commit latency and failures: slow or failed commits cause repeated processing and stall lag reduction.

  • Fetch latency: separates broker-side slowness from application-side delay.

  • Processing latency: time inside business logic, often the dominant part of end-to-end lag.

  • Dead-letter queue rate: a growing DLQ means poison messages are being rejected.

A consumer can be alive with healthy heartbeats while lag climbs, so combine liveness, throughput, lag and processing latency.

Storage, Retention and Compaction

Track disk usage per broker and partition, plus:

  • Log start offset: earliest retained offset.

  • Log end offset: newest appended offset.

  • Segment count and size: influences file handles, recovery time and compaction work.

  • Retention age: how long records stay available.

  • Compaction lag and tombstone count: matters for compacted topics keyed by current state.

  • Disk I/O latency and filesystem errors: often the earliest sign of storage trouble.

Sizing must include replicas, segment growth and bursts; hitting the filesystem limit can take a broker down and trigger leader elections.

Kafka Connect, Kafka Streams and Schema Registry

Component Metrics to watch
Kafka Connect Connector/task status, running tasks, sink throughput, offset-commit errors, transformation failures, retries, DLQ volume
Kafka Streams Processing latency, record-cache size, state-store size, commit latency, rebalance/restart counts, thread utilization, stream-thread error rate
Schema Registry Request latency, availability, compatibility-check failures, subject/version growth, HTTP error rate

A Connect sink task can look healthy while its destination is unavailable, so watch sink error and retry rates together. In Kafka Streams, large state stores or slow commits can cap throughput even when broker metrics look fine.

Operational Checks and Alerting

Collect broker, client, Connect, Streams and Schema Registry metrics through JMX or a supported exporter, and correlate them with CPU, disk latency, filesystem capacity, network drops and TLS/authentication failures. Two CLI checks are kafka-topics.sh --bootstrap-server broker1.example.com:9093 --describe --topic orders and kafka-consumer-groups.sh --bootstrap-server broker1.example.com:9093 --describe --group order-processors.

Alert on symptoms and trends, not isolated values: offline partitions, sustained ISR shrinkage, controller quorum loss, rapidly increasing lag, sustained produce/fetch errors, disk exhaustion, and repeated Connect or Streams task failures. Keep broker, client and consumer-owned metrics on separate dashboards so an application bottleneck is not blamed on Kafka.

61

How do you tune Apache Kafka for high throughput?

Tune Apache Kafka throughput by aligning partition parallelism, producer batching, consumer fetching, replication, and hardware with the workload. More partitions and larger batches can improve throughput, but hot keys, slow disks, excessive partitions, or weakened durability can erase those gains.

Read full answer

Applies toApache Kafka 4.x in KRaft mode with confluent-kafka 2.x

Primary Throughput Levers

Area Important settings Operational trade-off
Partitioning Topic partitions, consumer-group membership Partitions provide producer and consumer parallelism. Size them for target throughput and required consumers, not arbitrary scale. A hot key still overloads one partition.
Producers batch.size, linger.ms, compression.type Larger batches reduce request overhead; linger.ms waits briefly for more records. Compression reduces network and disk I/O but consumes CPU.
Consumers fetch.min.bytes, fetch.max.wait.ms, max.partition.fetch.bytes Larger prefetches reduce broker round trips, but increase latency and memory use. Very large records may require increasing max.partition.fetch.bytes.
Brokers num.network.threads, num.io.threads, replica.fetch.max.bytes Tune only after identifying CPU, request-queue, replication, or disk bottlenecks. Spread log.dirs across capable disks.
Durability acks=all, min.insync.replicas, idempotence Higher durability and replication consume additional network and disk bandwidth. acks=all does not itself provide exactly-once processing.

Kafka’s write path batches records by partition, compresses each batch, sends it to the partition leader, appends it locally, replicates it to followers, and acknowledges it according to the producer’s durability settings. Consumers fetch batches from assigned partitions and coordinate their progress through consumer-group offsets.

Delivery Guarantees and Ordering

  • Ordering: Kafka preserves record order within a partition, not across an entire topic. Use the partition key as the business ordering key.

  • Idempotent production: Set enable.idempotence=true to prevent duplicate records caused by producer retries. It does not prevent a consumer from processing a record again after a crash.

  • At-least-once consumption: Set enable.auto.commit=false, process the records, and then commit their offsets. A crash before the commit causes replay and possible duplicate side effects.

  • Exactly-once consumption: For consume-transform-produce within Kafka, use a Kafka transaction that includes output records and consumed offsets. Consumers reading transactional output must use isolation.level=read_committed. Kafka Streams processing.guarantee=exactly_once_v2 provides equivalent transaction semantics for Kafka records, state stores, and output. External database writes still require an atomic pattern such as an outbox.

  • Safe in-flight requests: With idempotence enabled, max.in.flight.requests.per.connection=5 preserves per-partition order while allowing parallel requests.

Production Operations and Failure Modes

  • Monitor produce and fetch latency, throughput, request queue time, GC pauses, disk latency and utilization, network saturation, under-replicated partitions, offline partitions, and frequent ISR changes.

  • Use replication factor 3 and min.insync.replicas=2 for important topics. A broker failure then retains write capacity, but losing additional replicas can stop writes.

  • Rebalances, oversized consumer batches, and slow processing can reduce effective throughput. Keep max.poll.interval.ms compatible with processing time and avoid committing offsets before processing succeeds.

  • Kafka Connect worker concurrency is controlled by tasks.max; normally use no more tasks than the connector’s partition count. Kafka Streams performance also depends on repartition topics, state-store throughput, and num.stream.threads.

  • The separate Confluent Schema Registry can centralize Avro and Protobuf schemas. Binary serialization can reduce payload size, and configured compatibility checks can block incompatible changes.

  • Use TLS with SASL_SSL or mTLS, least-privilege ACLs, and client quotas to limit noisy-neighbor impact.

  • Replication is not a backup. MirrorMaker 2 can provide cross-cluster replication and disaster recovery, but it is not a substitute for backups; use a separately tested backup and restore mechanism.

  • Upgrade brokers in a controlled rolling sequence while maintaining the KRaft controller quorum and adequate replica placement.

Code Example

Create a replicated topic; the partition count is an illustrative starting point that must be benchmarked:

bin/kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9093 \
  --create --topic orders \
  --partitions 24 \
  --replication-factor 3 \
  --config min.insync.replicas=2

Produce records with batching, compression, durable acknowledgements, and idempotence:

python -m pip install 'confluent-kafka~=2.0'
from confluent_kafka import Producer

errors = []

def delivery_report(error, message):
    if error is not None:
        errors.append(str(error))

producer = Producer({
    "bootstrap.servers": "kafka-1.example.com:9093",
    "acks": "all",
    "enable.idempotence": True,
    "compression.type": "lz4",
    "batch.size": 131072,
    "linger.ms": 20,
    "max.in.flight.requests.per.connection": 5,
    "delivery.timeout.ms": 120000,
    "on_delivery": delivery_report,
})

payload = b"x" * 1024
for sequence in range(10_000):
    producer.produce(
        "orders",
        key=f"order-{sequence}",
        value=payload,
    )

# flush() returns the number of messages still awaiting delivery.
remaining = producer.flush(30)
if remaining:
    raise RuntimeError(f"{remaining} Kafka message(s) were not delivered within 30 seconds")
if errors:
    raise RuntimeError("Kafka delivery failed: " + "; ".join(errors))

Tuning Workflow and Pitfalls

  1. Establish baseline throughput, p95/p99 latency, error rate, disk latency, and ISR stability.

  2. Benchmark in staging with representative record sizes, keys, compression, and consumer concurrency.

  3. Increase partitions and brokers only until useful parallelism is reached; changing one variable at a time makes regressions identifiable.

  4. Tune batching and prefetch sizes only after fixing skewed keys, undersized brokers, or slow disks.

  5. Avoid extremely large batches, frequent producer flushes, acks=0 for important records, manual offset commits before processing, and unbounded partition growth.

62

How can Apache Kafka be optimized for low-latency messaging?

Low-latency Kafka tuning minimizes the delay across producer batching, broker replication, consumer fetching, and application processing while preserving explicit durability and ordering requirements. The largest gains usually come from reducing unnecessary waits, preventing broker and consumer saturation, and measuring p95/p99 latency rather than optimizing averages.

Read full answer

Applies toApache Kafka 4.1 (KRaft mode), kafka-clients 4.1, Kafka Connect 4.1, and Kafka Streams 4.1

End-to-End Latency Model

A message’s end-to-end latency includes:

  1. Producer serialization and batching.

  2. Time spent waiting in linger.ms.

  3. Broker network processing and replication.

  4. Consumer fetch and poll latency.

  5. Application processing before the offset is committed.

Ordering is guaranteed only within a partition. Increase partition count to parallelize independent traffic, but excessive partitions increase controller, metadata, replication, and consumer-assignment overhead. Use keys when ordering related messages.

Broker and Storage Tuning

Tune from measurements rather than applying universal values:

Setting or resource Low-latency approach Trade-off
num.network.threads Increase if NetworkProcessorAvgIdlePercent is low More CPU usage and context switching
num.io.threads Increase for concurrent replication and disk I/O Benchmark against disk saturation
num.replica.fetchers Consider 2 or more on fast disks More concurrent disk reads
socket.send.buffer.bytes Test larger values such as 131072 Linux autotuning may already perform better
socket.receive.buffer.bytes Test larger values such as 131072 More memory per connection
fetch.min.bytes Use 1 for minimum fetch delay More frequent, smaller fetch requests
fetch.max.wait.ms Reduce from 500 to roughly 10-50 Greater broker and network load

Use local NVMe storage, keep Kafka data volumes separate from unrelated workloads, and preserve enough RAM for the OS page cache. Kafka normally relies on the page cache rather than forcing an fsync for every produce request.

Run the broker with Java 17 or newer. Keep the heap below roughly half the container’s memory so page cache remains available; a common starting point is a fixed 6–8 GiB heap for a moderately sized broker.

Producer and Consumer Configuration

Kafka 4.1 Java producer defaults include acks=all, idempotence enabled, and linger.ms=5. For latency-sensitive, high-throughput traffic:

  • Use batch.size around 32–64 KiB and linger.ms around 1–5 ms to batch requests without excessive waiting.

  • Use compression.type=lz4 when payloads are large enough for compression to reduce network and broker work.

  • Set buffer.memory according to the allowed batching concurrency; around 32–64 MiB is a reasonable starting point, not a universal limit.

  • Keep idempotence enabled so producer retries do not create duplicate records. This does not make downstream consumer processing exactly once.

  • acks=1 can reduce latency, but a leader failure can lose acknowledged records. Prefer acks=all with at least three replicas and min.insync.replicas=2 for production durability.

For consumers, use fetch.min.bytes=1 and a short fetch.max.wait.ms when minimum latency matters. Limit max.poll.records to prevent large processing batches, while keeping max.poll.interval.ms greater than the worst-case processing time. Use isolation.level=read_committed for transactional records; read_uncommitted is faster but can expose aborted data.

Consumer-side processing is normally at least once: commit offsets only after successful processing, and design handlers to tolerate duplicates. Exactly-once processing requires Kafka transactions, sendOffsetsToTransaction, and read_committed; it does not automatically cover non-Kafka side effects.

Connect, Streams, and Schema Registry

  • Kafka Connect: tune tasks.max, consumer.max.poll.records, producer.batch.size, and producer.linger.ms. Increasing offset.flush.interval.ms reduces commit overhead but affects recovery-point latency.

  • Kafka Streams: tune num.stream.threads for parallelism and commit.interval.ms against checkpoint overhead. Lower statestore.cache.max.bytes (even to 0) and commit.interval.ms when each update must be forwarded immediately; a larger cache reduces downstream churn but adds latency.

  • Schema Registry: use Avro or Protobuf rather than JSON where appropriate. Keep the client’s schema cache enabled to avoid fetching schemas on every request.

Code Example

Create and inspect a low-latency topic on three brokers:

BROKERS="kafka-1.example.com:9092"

kafka-topics.sh \
  --bootstrap-server "$BROKERS" \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config compression.type=lz4

kafka-topics.sh \
  --bootstrap-server "$BROKERS" \
  --describe \
  --topic orders

A latency-oriented Java producer:

import java.util.Properties;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;

public class Produce {
    public static void main(String[] args) throws Exception {
        Properties props = new Properties();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.example.com:9092");
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
        props.put(ProducerConfig.ACKS_CONFIG, "all");
        props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
        props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);
        props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864L);
        props.put(ProducerConfig.LINGER_MS_CONFIG, 2);

        try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
            producer.send(new ProducerRecord<>("orders", "key-42", "payload"))
                   .get(5, TimeUnit.SECONDS); // Synchronous for a latency benchmark
        }
    }
}

Production Operations

  • Monitor p95/p99 end-to-end latency, producer request latency, record-queue-time-avg, consumer fetch latency, consumer lag, NetworkProcessorAvgIdlePercent, RequestHandlerAvgIdlePercent, ISR changes, offline partitions, leader elections, GC pauses, disk latency, and CPU saturation.

  • Load-test with representative payload sizes, compression, TLS, replicas, and consumer concurrency. A loopback test is not representative.

  • Investigate long-tail latency through partition skew, hot keys, leader rebalancing, under-replicated partitions, page-cache pressure, and consumer rebalances.

  • Roll broker changes gradually, verify ISR after each restart, and never remove replicas before replacements have caught up.

  • Enable TLS and SASL where traffic crosses trust boundaries, restrict topics with ACLs, and include authentication overhead in latency testing.

  • Replication is not a backup. Mirror critical topics to another cluster or independent storage.

63

Why is Apache Kafka fast?

Kafka is fast because it combines sequential I/O, the operating system’s page cache, batching, compression, and selective zero-copy transfers. Its append-only partition logs let producers and consumers move data efficiently without database-style record updates or a dense index for every event.

Read full answer

Applies toApache Kafka 4.x

Architecture and I/O Path

  • Topics and partitions: A topic is divided into partitions. Each partition has one leader and optional follower replicas, allowing independent partitions to process traffic in parallel.

  • Append-only logs: Producers append batches to the leader partition. Consumers track positions with offsets instead of searching for or modifying individual records, keeping the hot read path inexpensive.

  • Replication: Followers replicate leader data. With acks=all, the leader waits for all current in-sync replicas to acknowledge the write, subject to min.insync.replicas.

  • KRaft: Kafka 4.x uses KRaft for controller quorum and metadata management; the ZooKeeper-based architecture is no longer supported.

  • Consumer groups: Group members receive disjoint partitions. Offsets can be committed automatically or, preferably for at-least-once processing, after successful business processing.

Why Kafka Is Fast

  • Sequential I/O: Each partition is an append-only log, so brokers perform sequential writes rather than random updates. Multiple partitions provide parallel I/O paths.

  • Page cache: The broker relies primarily on the OS page cache rather than keeping every record in the JVM heap. Data is handed to cached pages and flushed by the operating system; recently written data can also be served efficiently from cache.

  • Zero-copy: For plaintext transfers, Kafka can use kernel mechanisms such as sendfile, avoiding unnecessary user-space and kernel copies. This is not literally zero copying, and TLS adds CPU and copying costs.

  • Batching and compression: Producers group records and can compress them with lz4 or zstd. Fewer, larger requests reduce network overhead and improve disk efficiency.

  • Limited indexing: Offsets identify log positions directly. Kafka maintains sparse indexes where needed rather than a database-style index for every record.

Delivery Guarantees and Trade-offs

Model Configuration and behavior
At-most-once The consumer commits before processing; a crash can lose work.
At-least-once The consumer processes first and commits afterward; a crash can duplicate work.
Exactly-once within Kafka Requires a transactional producer with transactional.id, transactional offset submission, and consumers using isolation.level=read_committed. External side effects still require idempotency or distributed transaction coordination.

Ordering is guaranteed only within a partition, not across an entire topic. Idempotent producers ensure that retries of the same produce request do not create duplicate records, but idempotence alone does not provide end-to-end exactly-once processing.

Supporting Components

  • Kafka Connect: Source connectors read external systems, sink connectors write to them, and converters serialize or deserialize records. Record transformations are performed by Single Message Transforms (SMTs), while slow destinations or converters can become bottlenecks.

  • Kafka Streams: Processes streams and maintains local state, commonly using changelog topics for recovery and repartition topics when repartitioning is required. processing.guarantee=exactly_once_v2 enables Kafka-scoped exactly-once processing.

  • Schema Registry: Stores schemas and can reject schema-invalid records or enforce compatibility policies, while adding serialization and lookup overhead.

  • Retention and compaction: delete removes old records according to time or size. compact retains the latest record for each key, subject to tombstone and offset cleanup behavior. Neither is a backup mechanism.

Production Operations

  • Monitor produce and fetch latency percentiles, under-replicated partitions, ISR shrinkage, request queue time, disk usage, GC pressure, network saturation, and page-cache eviction.

  • Size partitions for expected throughput and consumer concurrency. A partition cannot process records in parallel, while excessive partitions increase controller, broker, and file-descriptor overhead.

  • Partition leaders should be distributed across brokers to avoid concentrating traffic on one machine.

  • Leader elections, unavailable replicas, and an insufficient ISR can temporarily reject writes or weaken the configured durability guarantees.

  • Use TLS, SASL authentication, and topic ACLs in production, accounting for TLS’s additional CPU and copying costs.

  • During rolling upgrades, preserve the replication factor and min.insync.replicas.

  • Kafka replication protects against broker failure, not corruption or deletion. Use an external backup or replication system and regularly test restoration procedures.

64

How do you size an Apache Kafka cluster?

Capacity-based Kafka cluster sizing starts with measured traffic, retention, replication, and availability requirements—not a fixed brokers-per-topic rule. The resulting design must sustain peak throughput, survive a broker failure without violating the SLA, and retain enough disk headroom for reassignments, compaction, and traffic spikes.

Read full answer

Applies toApache Kafka 4.x in KRaft mode

Sizing Inputs

Collect representative production data rather than relying only on averages:

  • Ingress rate: Peak messages per second and bytes per second for producers.

  • Message size: Include keys, headers, and value distributions; large records require fewer messages but more network, heap, and I/O.

  • Retention: Estimate both time-based retention and any retention.bytes limit.

  • Replication factor: Usually 3 for critical production topics.

  • Consumer behavior: Number of consumer groups, maximum parallelism, expected lag, and how quickly processing must catch up.

  • Availability target: Determine how many brokers, replicas, and KRaft controllers must remain available.

  • Compression and batching: compression.type, producer batch.size, and linger.ms materially affect throughput.

For compacted topics, size for churn, the active key set, and old record versions awaiting compaction rather than assuming data disappears immediately. Tombstones remain until compaction occurs and the topic’s delete.retention.ms window has elapsed; interrupted compaction can leave old records temporarily on disk.

Architecture and Partition Sizing

A broker stores topic partitions. Each partition has one leader and optional follower replicas; consumers in the same consumer group divide partitions among themselves. Therefore:

  • Partition count determines producer and consumer parallelism.

  • Ordering is guaranteed only within one partition.

  • A topic with 12 partitions and replication factor 3 creates 36 partition replicas.

  • Increasing partitions permits more consumers, but adds metadata, file handles, replication traffic, and rebalance work.

Choose the partition count from benchmarked per-partition throughput, not an arbitrary rule of thumb. A useful initial broker estimate is:

minimum brokers = ceil((partitions × replication factor) / planned replicas per broker)

Benchmark with realistic record sizes, compression, acknowledgements, and replica placement. The design must still meet peak traffic after one broker fails. For high availability, use three KRaft controllers across failure domains; controllers should normally run on dedicated nodes even though they consume less CPU and memory than brokers.

Storage and Network Capacity

First calculate logical retained data:

retained bytes = messages per second × average uncompressed message bytes × retention seconds

Then calculate replicated storage:

disk required = retained compressed bytes × replication factor × segment/index overhead

Divide that requirement by the target maximum disk utilization. Keeping brokers around 60–70% utilized is a practical starting point, leaving space for segment creation, partition reassignment, temporary retention overlap, and traffic spikes.

Also budget:

  • Interbroker network: Replication and follower fetches, especially during reassignment or recovery.

  • Client network: Producer ingress and consumer egress; replicated storage does not multiply client ingress by the replication factor.

  • Request handling: Partition requests, compression CPU, authentication, and TLS overhead.

  • Internal topics: Consumer offsets, transaction state, and schema or Connect-related state.

  • Headroom: At least enough capacity to recover after a failure without exceeding disk safety limits.

Kafka Connect worker pools, Kafka Streams instances with state stores, and Schema Registry clusters have separate CPU, memory, disk, and availability requirements. Use dedicated infrastructure when their workload could destabilize the broker cluster.

Code Example

Create and inspect a replicated topic with a durable ISR threshold:

bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000

bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe \
  --group order-processors

bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe \
  --under-replicated-partitions

The 12 partitions must be validated against measured throughput and consumer concurrency. min.insync.replicas=2 with replication factor 3 allows writes while one replica is unavailable, assuming at least two replicas remain in the ISR.

Production Failure Modes and Operations

  • Delivery semantics: acks=all with enable.idempotence=true protects acknowledged writes from producer retry duplicates. Consumer-side at-least-once processing can still duplicate work if a consumer commits its offset before processing completes.

  • Exactly-once processing: Use Kafka transactions and isolation.level=read_committed, or Kafka Streams processing.guarantee=exactly_once. External databases require transactional integration or idempotent consumers.

  • Monitoring: Alert on under-replicated and offline partitions, ISR changes, consumer lag, request latency, broker disk utilization, GC pauses, network saturation, and producer errors.

  • Broker failure: With three replicas and two required ISR replicas, one broker loss should not stop writes, but leader elections and re-replication still create temporary latency.

  • Partition growth: Use bin/kafka-topics.sh --alter --topic <topic> --partitions <new-count> to increase partitions; use kafka-reassign-partitions.sh to redistribute partitions or replicas in a controlled operation.

  • Upgrades: Follow Kafka’s version-specific rolling-upgrade procedure, maintain controller quorum, and verify replication health after each stage.

  • Disaster recovery: Replication is not a backup. Use a geographically separate cluster, typically with MirrorMaker 2, and test recovery objectives.

  • Security: Use TLS in transit, SASL authentication, least-privilege ACLs, encrypted disks, and controlled certificate rotation. Place brokers across separate failure domains and avoid exposing plaintext listeners outside the trusted network.

65

How does Apache Kafka redistribute partition replicas between brokers?

Apache Kafka redistributes partition replicas through partition reassignment: the controller records a new replica assignment, new replicas copy the partition’s log until they catch up, and obsolete replicas are removed. This is separate from a consumer group rebalance, which assigns topic partitions to consumer instances.

Read full answer

Applies toApache Kafka 3.6–4.x in KRaft mode

Core Concept

A Kafka partition has one leader and usually one or more follower replicas. The leader serves client requests, while followers maintain copies for fault tolerance and potentially read scaling.

Kafka does not automatically move existing partition replicas when brokers are added or removed. An operator requests reassignment when decommissioning a broker, expanding capacity, changing rack placement, repairing replica placement, or redistributing disk load.

Operation What changes Typical trigger
Partition reassignment Replica placement and data location Broker replacement or capacity rebalancing
Leader election Which replica is the leader Preferred-leader election or broker failure
Consumer group rebalance Which consumers own partitions Consumer membership or partition-assignment changes

The logical partition and its records remain unchanged during reassignment. For example, orders-3 continues to contain the same records; only the brokers storing its replicas change.

Under-the-Hood Sequence

  1. The controller validates the requested broker IDs and replica-assignment constraints.

  2. The new assignment is recorded in the KRaft metadata log.

  3. The Kafka replica fetcher copies records from the current leader, or from a subsequently elected leader if leadership changes.

  4. The new replica catches up to the leader’s log-end offset and joins the in-sync replica set, or ISR.

  5. Once the replacement is caught up, the obsolete replica is removed from the partition’s replica set.

  6. An operator can optionally request a preferred-leader election after the target leader is in the ISR. The preferred leader is the first broker in the partition’s current replica list.

Replica movement is asynchronous. During the transition, a partition can temporarily contain more replicas than its configured replication factor because Kafka adds the replacement before removing the obsolete replica. Replication traffic can affect client latency, so leader and follower replication throttles can constrain migration speed.

Operational Workflow

The administrative workflow is implemented by kafka-reassign-partitions.sh. The operator supplies a version 1 assignment document containing the topic, partition, target replica broker IDs, and target log directories, then applies it with the tool’s execution mode.

The controller expands the replica set and waits for the new replica to catch up before contracting the set. Administrators inspect the partition with kafka-topics.sh --describe and verify the reassignment with the verification mode of kafka-reassign-partitions.sh. A preferred-leader change can then be requested with kafka-leader-election.sh when required.

Failure Modes and Monitoring

  • Insufficient destination capacity: The new replica cannot catch up, so reassignment remains incomplete.

  • ISR degradation: If too many replicas leave the ISR, writes can fail when min.insync.replicas cannot be satisfied.

  • Data loss risk: Keep unclean.leader.election.enable=false; an out-of-sync replica must not be promoted automatically.

  • Leader movement: Leadership changes can cause transient request retries or brief unavailability.

  • Consumer impact: Consumers normally retain the same partition assignments. Replica movement can cause metadata updates and transient retries, but it does not by itself require a consumer-group rebalance.

  • Monitoring: Track UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflineReplicaCount, leader-change rate, request latency, disk utilization, replication throughput, and consumer lag.

Production Best Practices

  • Preserve redundancy: For critical topics, commonly use replication.factor=3 and min.insync.replicas=2, adjusted to the required failure-domain tolerance.

  • Plan capacity: Ensure destination brokers have enough disk space and network capacity in their configured log.dirs.

  • Use rack awareness: Set broker.rack and distribute replicas across failure domains.

  • Throttle migrations: Use leader.replication.throttled.replicas and follower.replication.throttled.replicas carefully, because these broker-level settings can affect multiple partitions.

  • Decommission safely: Reassign partitions, verify completion and ISR health, and only then stop the broker.

  • Maintain independent backups: Replication does not protect against accidental deletion, malformed writes, or retention expiry; use cross-cluster replication or snapshots when independent recovery is required.

  • Secure administration: Use TLS and SASL, restrict administrative access, and provide protected client properties to administrative tools with --command-config.

66

How can an Apache Kafka cluster be upgraded without downtime?

A no-downtime Kafka upgrade uses a rolling upgrade: replace one node at a time while preserving replication, KRaft quorum, and client connectivity. The cluster remains available, although leader elections can cause short latency increases and retriable client errors.

Read full answer

Applies toApache Kafka 4.1.x with KRaft; rolling paths are 4.0 to 4.1 and 3.9 to 4.0 to 4.1

Upgrade Strategy

  • Upgrade one node at a time: Never restart multiple brokers or controllers simultaneously. Keep enough replicas and voting controllers online to preserve quorum and partition leadership.

  • Upgrade KRaft controllers first: Upgrade the controller quorum one controller at a time, then upgrade broker-only nodes. For combined broker/controller nodes, upgrade them individually while retaining at least two of three voters.

  • Follow the compatibility path: Upgrade directly only across officially supported versions. A 3.9 cluster must first be upgraded to 4.0 before proceeding to 4.1.

  • Leave ZooKeeper clusters first: Kafka 4.x does not support ZooKeeper. A ZooKeeper-based cluster must be migrated to KRaft using a 3.9 release before it can enter the Kafka 4.x rolling-upgrade path.

  • Upgrade clients afterward: Keep existing clients running during the broker rollout, then upgrade Connect workers, Kafka Streams applications, Schema Registry instances, and other clients gradually.

  • Preserve rollback compatibility: Do not advance the message-format version or enable new KRaft metadata features until every node is upgraded and the rollback window has closed.

Pre-Upgrade Safeguards

Area Recommended condition
Controllers Three or more KRaft voters; restart only one at a time
Topic replication Replication factor of at least 3
Write durability min.insync.replicas=2 with producer acks=all
Leader safety unclean.leader.election.enable=false
Internal topics offsets.topic.replication.factor=3, transaction.state.log.replication.factor=3, transaction.state.log.min.isr=2
Capacity At least 50% free disk and sufficient network headroom
Recovery Metadata quorum snapshot, configuration and ACL backups, plus replicated partition data

Before starting, confirm that there are no under-replicated or offline partitions, disk errors, or abnormal consumer lag. Use kafka-broker-api-versions.sh to inspect broker-supported protocol versions; obtain connected client versions from application inventory or client metrics instead.

Rolling Upgrade Procedure

  1. Record the current binary version, broker configuration, KRaft node.id, listeners, security settings, and voter configuration such as controller.quorum.voters for static quorums.

  2. Create a consistent KRaft metadata snapshot using kafka-metadata-quorum.sh snapshot; also back up broker and topic configurations and ACLs. Use a secondary cluster or equivalent replication for partition-data disaster recovery rather than relying on a metadata snapshot alone.

  3. Stop one node cleanly and install the target Kafka release while preserving its data directories and configuration. Do not reformat an existing KRaft storage directory.

  4. Ensure controlled.shutdown.enable=true so leadership can be moved before shutdown.

  5. Point the service definition at the new installation, start the upgraded node, and wait for it to rejoin normally.

  6. Verify controller quorum, partition leadership, ISR status, consumer lag, broker health, and client error rates before upgrading another node.

  7. Upgrade the KRaft controllers first, one at a time, and then broker-only nodes. If roles are combined, maintain controller quorum throughout.

  8. After all brokers are upgraded, roll Connect workers, Kafka Streams applications, Schema Registry instances, and clients gradually, validating each stage.

Validation and Client Behavior

  • acks=all with min.insync.replicas=2 prevents acknowledged writes from being accepted with weaker replication; unavailable writes may block instead.

  • Ordering is guaranteed only within a partition.

  • Producer idempotence prevents duplicate records from producer retries within the producer’s retry and session guarantees.

  • Kafka read-process-write exactly-once processing requires a transactional producer with a unique transactional.id, enable.idempotence=true, and consumers using isolation.level=read_committed. Kafka Streams additionally uses processing.guarantee=exactly_once. This does not make arbitrary external side effects exactly-once.

  • Consumers should use bounded exponential backoff for retriable errors such as NOT_LEADER_OR_FOLLOWER and REQUEST_TIMED_OUT.

Code Example

Run these commands from a host containing the target Kafka distribution:

BROKER=kafka-1.example.com
BROKER_ID=1
BROKER_SERVICE=kafka.service

# Check the current KRaft quorum state.
./bin/kafka-metadata-quorum.sh \
  --bootstrap-server "$BROKER:9092" describe --status

# List under-replicated partitions.
./bin/kafka-topics.sh \
  --bootstrap-server "$BROKER:9092" \
  --describe --under-replicated-partitions

# Inspect consumer-group assignments and lag.
./bin/kafka-consumer-groups.sh \
  --bootstrap-server "$BROKER:9092" \
  --describe --group payments

# Inspect APIs supported by the brokers; this does not report client versions.
./bin/kafka-broker-api-versions.sh \
  --bootstrap-server "$BROKER:9092"

# Save the broker's dynamic configuration.
./bin/kafka-configs.sh \
  --bootstrap-server "$BROKER:9092" \
  --describe --entity-type brokers --entity-name "$BROKER_ID" --all

# Restart only this broker after pointing the unit at the new installation.
sudo systemctl restart "$BROKER_SERVICE"

Repeat the validation commands after each restart and before proceeding.

Failure Modes and Rollback

  • Reduced redundancy: Stopping one node removes a replica and may trigger leader elections. Stop if remaining replicas cannot satisfy min.insync.replicas.

  • Quorum loss: Restarting two of three controllers simultaneously can make the cluster unavailable.

  • Disk pressure: Log growth, compaction, and replica lag can consume space. Monitor log-dir-size, I/O latency, and request queues.

  • Incompatible metadata: Do not assume binary downgrade is safe after advancing message-format or KRaft metadata features. Roll back before such changes when the release documentation permits it.

  • Security regressions: Preserve TLS and SASL listener configuration, rotate certificates beforehand, and ensure upgraded broker identities remain valid.

  • Connector disruption: Connect worker replacement can restart tasks and replay records. Validate sink idempotency and monitor duplicate handling.

“No downtime” means continuous service availability, not zero request latency. Capacity planning, redundancy, staged validation, and tested rollback procedures are essential.

67

How does MirrorMaker 2 replicate Kafka data across clusters?

MirrorMaker 2 is Apache Kafka’s replication system for moving records between independent Kafka clusters. It runs on Kafka Connect, preserves ordering within each source partition, and normally provides at-least-once replication for disaster recovery and active-active use cases.

Read full answer

Applies toApache Kafka 3.9 and 4.x

Architecture and Flow

MirrorMaker 2 uses a distributed Connect worker and the MirrorSourceConnector:

  • Source replication: The connector consumes records from source-cluster topics, creates corresponding target topics when necessary, and produces the records to the target cluster.

  • Heartbeat connector: MirrorHeartbeatConnector emits heartbeat records used for cross-cluster lag measurement and replication-cycle detection.

  • Checkpoint connector: MirrorCheckpointConnector stores source topic offsets in a target-cluster checkpoint topic so replication can resume after a restart.

  • Offset-sync connector: MirrorOffsetSyncConnector optionally synchronizes consumer-group offsets. The offset-sync topic can be located in either the source or target cluster.

For example, orders.created in cluster A is replicated to orders.created in cluster B. The default replication policy normally preserves the source partition count and maps a source partition to the corresponding target partition.

Model Behavior Operational trade-off
Active-passive Cluster B receives replicated data but normally accepts no application writes until failover Simplest recovery model; consumers must be redirected during promotion
Active-active Applications write to both clusters Requires strict topic, consumer-group, and loop-prevention policies; cross-cluster ordering cannot be guaranteed

Kafka 4.x clusters use KRaft, but MirrorMaker 2 communicates through Kafka client and data APIs and does not directly access the controller quorum.

Replication and Consumer-Group Behavior

  • Ordering: Records are produced in source-partition order to the mapped target partition. There is no global ordering across partitions or topics.

  • Delivery: Replication is at-least-once. A failure after the target accepts records but before the source connector commits its offset can cause duplicates, so consumers should be idempotent or use deduplication.

  • Exactly once: MirrorMaker 2 does not provide transactional exactly-once processing across independent clusters by itself. End-to-end exactly-once behavior requires an application-specific transactional and deduplication design.

  • Consumer-group failover: With offset synchronization enabled, consumer-group offsets can be made available for use in the secondary cluster. During failover, stop writes to the old active cluster, confirm that replication and offsets are current, and then redirect consumers to the promoted cluster.

  • Active-active caution: Heartbeats prevent immediate feedback loops, but application writes can still create logical loops. Use coordinated topic mappings and separate application group identifiers where appropriate.

Key Configuration

Important worker and connector options include:

  • source.cluster.alias and target.cluster.alias: Identify the clusters configured on the Connect worker.

  • connector.class: Set to org.apache.kafka.connect.mirror.MirrorSourceConnector.

  • topics: Regular expression selecting the topics to replicate.

  • replication.policy.class: Selects the policy governing topic and partition mapping and replication-cycle handling.

  • offset-syncs.topic.location: Selects the cluster in which synchronized consumer offsets are stored.

  • sync-group.connector.enabled: Enables consumer-group offset synchronization.

  • sync-group.connector.groups: Restricts synchronization to matching consumer-group names.

  • heartbeat.interval.ms: Controls heartbeat frequency and therefore lag-detection resolution.

MirrorMaker 2 replicates records, not other cluster metadata. Kafka Streams state stores, Schema Registry schemas, ACLs, and topic configuration require separate management. Consumer-group offsets can optionally be synchronized, but group assignments, in-flight work, and other consumer state are not replicated. The target log is compacted normally according to the target topic’s own compaction configuration.

Failure Modes and Security

  • Source unavailable: The connector remains retryable and resumes from its last committed position when source data becomes available again, subject to source retention.

  • Target unavailable: Producers retry until delivery limits are reached, after which the Connect task fails and the framework retries it. Monitor producer errors and lag; do not delete checkpoint topics as a recovery mechanism.

  • Consumer lag: A replicated target can still have high consumer lag because replication throughput does not guarantee downstream processing capacity.

  • Data loss or corruption: Replication is not a complete backup. It can propagate deletions or corruption, and recovery depends on target retention and integrity.

  • Security: Use TLS and SASL for cluster connections. The source identity needs read access to replicated topics and permissions needed by offset synchronization; the target identity needs write access to replicated records and permission to create or write heartbeat and checkpoint topics. If offset-sync storage is enabled in the source, that cluster also needs the corresponding topic permissions.

  • Connect REST access: Restrict connector creation and configuration changes to administrators, and expose the REST API only through appropriate network controls and authentication.

Production Practices

Monitor connector and task state, retries, producer errors, heartbeat-based replication lag, target consumer lag, broker latency, disk usage, under-replicated partitions, and leader elections. Back up both Kafka clusters and the Connect configuration, offset-storage, and status-storage topics using appropriate Kafka backup procedures.

Upgrade the source and target brokers according to Kafka’s supported rolling-upgrade sequence, then upgrade the Connect workers running MirrorMaker 2 and verify connector status after each worker change. Because MirrorSourceConnector is single-task, scale large workloads by increasing source-topic partition counts or deploying separate connector instances with disjoint topic selections rather than relying on tasks.max alone. Ensure target brokers, retention, and consumer capacity can absorb the replication peak during failover.

68

How do you design Apache Kafka for disaster recovery?

A resilient Kafka disaster-recovery design combines multi-AZ replication, independent backups or a secondary region, and explicit RPO/RTO targets. Broker replication survives hardware and zone failures, but not regional failure, deletion, corruption, or ransomware.

Read full answer

Applies toApache Kafka 4.x with KRaft and confluent-kafka 2.x

Failure Model and Recovery Objectives

Failure Primary protection Effect
Broker or disk failure In-sync replicas Automatic failover; acknowledged data survives if enough replicas remain
Availability-zone failure RF 3 across three zones Service continues with reduced capacity
Region failure MirrorMaker 2, dual writes, or another independent copy Fail over or redirect clients
Deletion or corruption Immutable, versioned backups Restore selected topics or the cluster
Ransomware or credential compromise Separate credentials and immutable backups Recover into a clean environment

RPO is the maximum acceptable data loss; RTO is the maximum acceptable recovery time. Multi-region replication normally has an RPO equal to observed replication lag, not zero.

Data Durability Architecture

  • Run at least three brokers across three zones. In combined mode, spread a three-controller quorum across zones; in dedicated mode, run at least three controllers separately from the brokers.

  • Use replication factor 3 and min.insync.replicas=2 for critical topics. Keep unclean.leader.election.enable=false to prevent data loss during an unclean election.

  • Configure producers with acks=all and enable.idempotence=true. Idempotence prevents duplicates from producer retries, but it is not a backup.

  • Kafka preserves ordering only within a partition. At-least-once consumers must process a record before committing its offset.

  • Kafka’s exactly-once read-process-write semantics for Kafka records require transactional producers, a stable transactional.id, isolation.level=read_committed consumers, and a suitably replicated transaction-state log. External side effects still require idempotency or application-specific coordination.

  • Protect Kafka Streams state-store changelogs and application offsets. Kafka Connect configurations and sink idempotency must also be considered because sink writes can be repeated.

Backup and Restore

Back up broker data directories, metadata.log.dir if separate, configuration, ACLs, and the original cluster ID. Store encrypted, immutable copies with separate credentials. Do not copy live broker files without a supported snapshot or consistency mechanism.

For a new KRaft cluster, initialize every node with the same cluster ID and node-specific configuration containing a unique node.id:

CLUSTER_ID=$(./bin/kafka-storage.sh random-uuid)
./bin/kafka-storage.sh format \
  --cluster-id "$CLUSTER_ID" \
  --config server.properties

During restoration, restore data and metadata consistently. Do not re-format restored metadata or change the cluster ID.

A recovery sequence is:

  1. Restore brokers into an isolated network.

  2. Start a controller quorum and verify it with kafka-metadata-quorum.sh.

  3. Restore remaining brokers gradually and check partition leadership.

  4. Validate ISR, topic metadata, consumer offsets, and representative records.

  5. Reconnect applications only after completeness checks pass.

Test restores regularly. An untested backup is only an assumption.

Cross-Region Recovery

  • MirrorMaker 2: Asynchronously replicates topics and consumer offsets. Failover requires changing bootstrap addresses or routing outside Kafka.

  • Active-active dual writes: Publish equivalent records to both regions. This reduces lag-related loss but requires idempotent consumers or reconciliation for partial writes.

  • Replicate Schema Registry schemas and compatibility policies separately; topic replication does not copy Schema Registry metadata.

  • Region-specific consumer-group IDs simplify independent operations. Reusing IDs across independently operated clusters requires an explicit offset-replication and failover policy.

  • Test bandwidth, encryption, replication lag, partial writes, and ordering under production load.

Code Example

Create a replicated topic and explicitly enforce its durability settings:

./bin/kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --create --if-not-exists \
  --topic orders \
  --partitions 12 \
  --replication-factor 3

./bin/kafka-configs.sh \
  --bootstrap-server kafka.example.com:9092 \
  --alter \
  --entity-type topics \
  --entity-name orders \
  --add-config min.insync.replicas=2,unclean.leader.election.enable=false

A Python producer using idempotence and acknowledgements from all in-sync replicas:

from confluent_kafka import Producer

def delivery_report(err, msg):
    if err is not None:
        print(f"Delivery failed: {err}")
    else:
        print(f"Delivered to {msg.topic()}[{msg.partition()}]")

producer = Producer({
    "bootstrap.servers": "kafka.example.com:9092",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "orders-producer",
    "sasl.password": "password",
    "enable.idempotence": True,
    "acks": "all",
})

producer.produce(
    "orders",
    key=b"order-123",
    value=b'{"status":"paid"}',
    on_delivery=delivery_report,
)
producer.flush(10)

Production Practices

  • Monitor under-replicated and offline partitions, leader elections, disk usage, request latency, consumer lag, replication lag, backup age, and restore-test age.

  • Use TLS, SASL, least-privilege ACLs, administrative MFA, secret rotation, and separate backup credentials.

  • Upgrade brokers and clients in rolling stages while watching ISR, latency, and error rates. Keep recovery procedures compatible with the Kafka version.

  • Exercise broker loss, zone loss, regional failover, offset restoration, and backup restoration through game days.

  • Automate infrastructure while treating Kafka metadata and cluster identity as controlled recovery artifacts.

69

What is Apache Kafka tiered storage?

Apache Kafka tiered storage keeps hot partition segments on the broker’s local disks and moves older, closed segments to a remote object store, while producers, consumers and consumer groups keep using the same APIs, the same offsets and the same replication model.

Read full answer

Applies toApache Kafka 4.x (tiered storage was early access in 3.6 and became production-ready in 3.9) with a plugin implementing the Kafka remote-storage SPIs

How It Works

Kafka partitions are divided into log segments. During normal operation:

  1. Producers append new records to the active segment in the partition leader’s local tier.

  2. When a segment rolls over, the broker uploads it to the remote tier through a remote-storage plugin that implements the RemoteStorageManager and RemoteLogMetadataManager SPIs.

  3. The broker records remote-log metadata for the segment — its offsets, timestamps, size and remote location — in the metadata manager.

  4. After the upload is confirmed, the local copy is kept only until it falls outside the topic’s local.retention.ms / local.retention.bytes; then it is deleted locally, while overall retention.ms / retention.bytes govern the remote copy.

  5. Consumers keep using the same partition and offset. When a consumer needs data that is no longer local, the broker fetches it from the remote tier and streams it to the client; remote data is not written back into the local log, though the remote-segment index files are still stored locally.

Tier Typical contents Trade-off
Local tier Active segments, indexes, remote-log index files Low latency and high throughput, but expensive capacity
Remote tier Older closed segments Lower storage cost, but higher read latency and network dependency

Tiered storage is not a replacement for replication: normal leader/follower replication still applies, and the plugin is responsible for safely uploading, retrieving and deleting remote segments.

Enabling It

Apache Kafka ships the framework, but no official object-store implementation; you must supply a plugin whose classes implement the remote-log SPIs.

On each broker that will host partitions of the topic, set remote.log.storage.system.enable=true in the broker properties. This is a read-only broker setting, so it requires a broker restart. Point remote.log.storage.manager.class.name (and, if you do not use the built-in topic-based metadata manager, remote.log.metadata.manager.class.name) at the classes provided by your plugin, and make sure the plugin jars are on the class path of every broker involved.

On the topic, set the topic-level config remote.storage.enable=true — either at creation time with the --config option of kafka-topics.sh, or afterwards with kafka-configs.sh using --alter --add-config against the topic entity. Confirm it with kafka-configs.sh --describe for the topic. Before Kafka 3.9 tiered storage could not be turned off for a topic; from 3.9 it can be disabled, but doing so requires remote.log.delete.on.disable=true, which deletes the topic’s remote data, so plan the data lifecycle before enabling it. Compacted topics cannot use tiered storage. Enabling it on an existing topic means the broker will upload its existing closed segments, so plan for that I/O.

Consumer and Delivery Semantics

Tiered storage does not change Kafka’s fundamental guarantees:

  • Ordering: records are ordered within a partition, not across partitions.

  • At-least-once delivery: consumers commit offsets after processing and may reprocess records after a failure.

  • Exactly-once processing: requires Kafka transactions, or the appropriate idempotent producer and read-committed consumer configuration; it is not conferred by moving data to remote storage.

  • Consumer groups: offsets remain managed by the consumer-group protocol; internal topics such as __consumer_offsets are not tiered by this feature.

  • Latency: a remote read adds fetch latency on cache misses, so consumers reading old data may see slower fetches and temporarily higher lag.

Production Operations and Failure Modes

  • Upload failures: investigate plugin errors, object-store throttling, expired credentials and network reachability; monitor upload lag and the progress of segment checkpoints.

  • Remote read failures: track fetch latency, error rate and consumer lag; a broker must be able to reach the object store to serve remote segments.

  • Local disk pressure: monitor broker disk usage including active segments, the local cache of remote-log index files (sized by remote.log.index.file.cache.total.size.bytes).

  • Data durability: use object-store versioning, provider-side replication, lifecycle policies and an independent backup strategy where the retention requirement demands it.

  • Metadata recovery: protect KRaft metadata separately from the remote segment data; restoring objects without the matching Kafka metadata and topic configs can leave partitions unreadable.

  • Upgrades: upgrade broker and plugin together, check SPI compatibility, and rehearse remote reads, leader failover, broker replacement and cleanup before rolling out.

Security and Trade-offs

Use TLS and SASL for client-to-broker traffic, and secure broker-to-object-store traffic with private endpoints, IAM roles or narrowly scoped credentials. Never put storage credentials in topic configuration or on command lines visible in shell history and logs; rely on the provider’s encryption controls for data at rest.

The benefit is lower storage cost and better use of local disks. The costs are remote-fetch latency, a hard dependency on the storage provider, extra operational complexity and possible cleanup or consistency issues in the plugin. For latency-sensitive topics, keep hot data local longer and use tiered storage mainly for segments that are read rarely.

Best Practices and Common Pitfalls

  • Start with a non-critical topic and measure upload throughput, remote-read latency and local-disk savings before rolling out.

  • Size the local tier for active data, indexes and compaction work, not just active partitions.

  • Test leader failover and broker replacement while remote segments are being read.

  • Ensure the plugin implements durable deletion and retention cleanup correctly; orphaned objects cost money and complicate recovery.

  • Do not assume tiering changes consumer offsets, transaction semantics, partition ordering or replication guarantees.

  • Keep a tested restore procedure covering object-store data, Kafka metadata, topic configs and plugin configuration.

70

Self-managed Kafka vs Confluent Cloud, Amazon MSK, and Redpanda

Self-managed Apache Kafka provides maximum control over infrastructure, deployment, upgrades, and cost, while managed Kafka services offload broker operations. Confluent Cloud and Amazon MSK are managed Kafka offerings; Redpanda is a Kafka-API-compatible alternative with a different internal architecture and ecosystem coverage.

Read full answer

Applies toApache Kafka 4.x; current Confluent Cloud, Amazon MSK, and Redpanda managed offerings

Architecture and Ecosystem

Area Self-managed Kafka Confluent Cloud Amazon MSK Redpanda
Deployment VMs, containers, Kubernetes with Strimzi, or bare metal Vendor-operated Kafka clusters AWS-managed provisioned or serverless clusters Single-binary, Kafka-compatible service using Raft
Control plane KRaft controllers in current Kafka 4.x deployments Fully vendor-managed Managed; KRaft is available for supported provisioned configurations Redpanda’s Raft implementation, not Kafka KRaft
Availability Designed by the customer Region and tier dependent AWS availability-zone and deployment dependent Vendor-managed regions and cluster configurations
Kafka Connect Full Kafka distribution support Native Confluent Connect MSK Connect with supported source and sink connectors Redpanda Connect; connector and configuration compatibility must be validated
Kafka Streams Supported by Apache Kafka Supported and managed by Confluent Supported, subject to client compatibility Kafka API compatibility exists, but support and feature parity are not identical to Apache Kafka
Schema Registry Separate Confluent Schema Registry deployment Integrated managed service Usually integrated through supported registry or external tooling Built into the Redpanda broker binary (Confluent-compatible API)
Operations Customer provisions, monitors, patches, scales, and upgrades brokers Vendor handles broker lifecycle and most infrastructure work AWS handles broker lifecycle and infrastructure work Vendor handles the managed service; customers still tune topics, clients, and access

Self-managed Kafka is not automatically cheaper or more reliable. Its flexibility increases operational responsibility, especially when managing broker failures, disk capacity, controller quorum, network isolation, and rolling upgrades.

Delivery Guarantees

Managed services do not change Kafka’s fundamental delivery model:

  • Ordering: Records are ordered within a partition, not across an entire topic. A key routed consistently to one partition preserves ordering for that key.

  • At-most-once delivery: Producers avoid retries and acknowledge processing before persistence. It can lose records after a producer failure.

  • At-least-once delivery: Producers retry failed sends, and consumers commit offsets after processing. Duplicate records are possible, so handlers should be idempotent.

  • Exactly-once processing: Kafka’s transactional model can atomically publish records, consume input records, and commit consumer offsets within Kafka.

For transactional Kafka workloads, use enable.idempotence=true, acks=all, a unique stable transactional.id per producer instance, transactional send-and-offset APIs, and isolation.level=read_committed on consumers. Merely enabling idempotence does not make external database writes exactly-once; use an outbox pattern, idempotent writes, or another coordination mechanism.

Production Operations

For self-managed Kafka, monitor broker disk utilization, request latency, network throughput, under-replicated and offline partitions, controller health, JVM pressure, and consumer lag. An exact topic inspection command in a standard Kafka installation is:

/opt/kafka/bin/kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders

Key production concerns include:

  • Capacity planning: Add brokers before disks fill. A partition is assigned to one broker, so partition count affects rebalance and recovery time.

  • Hot partitions: A popular key can overload one partition or broker. Review partition-level traffic rather than only cluster averages.

  • Consumer lag: Lag can indicate slow processing, a rebalance, a connectivity problem, or an undersized consumer group.

  • Rebalances: Long processing cycles can exceed max.poll.interval.ms and cause repeated partition revocation.

  • Configuration: min.insync.replicas=2, replication factor 3, and acks=all commonly prevent acknowledged records from being lost with one broker failure. This reduces write availability during larger failures.

  • Retention: retention.ms controls time retention, while cleanup.policy=compact retains the latest value for each record key. Compaction does not immediately remove every older duplicate.

  • Upgrades: Test client, broker, Connect, and Schema Registry compatibility. Upgrade brokers in a controlled rolling sequence and verify controller quorum health.

Managed platforms remove broker patching and infrastructure failures from the customer’s responsibility, but they do not remove the need to monitor lag, errors, quotas, partition skew, schema compatibility, and client configuration.

Security and Failure Modes

Use TLS and authenticated protocols such as SASL/SCRAM or SASL/OAuth where supported, restrict broker access with network controls, and apply least-privilege ACLs or role-based access. Managed services may provide IAM integration, private networking, and key-management options, but application credentials and authorization design remain customer responsibilities.

Replication is not a backup: corruption, accidental deletion, or faulty producers can be replicated too. Define retention, cross-region recovery, and backup procedures explicitly. MirrorMaker 2 or a managed disaster-recovery product can protect against regional failure, but replication latency and operational complexity must be included in the recovery plan.

Selection Guidance

Choose self-managed Kafka when strict infrastructure control, custom networking, predictable high utilization, or existing Kafka expertise justify operating the platform. Choose Confluent Cloud when the broader Confluent ecosystem and vendor-managed operations matter most. Choose Amazon MSK for tight AWS integration, provisioned or serverless consumption models, and AWS networking and governance. Choose Redpanda or Redpanda Cloud when a simpler Kafka-compatible architecture is attractive, but validate connector, transaction, schema-management, and client compatibility before migration.

71

How do you test Kafka-based applications reliably?

Integration testing against real Kafka brokers is the core of testing Kafka-based applications: layer it with unit tests, contract tests, failure injection and end-to-end tests so application logic and infrastructure-dependent behaviour are both covered.

Read full answer

Applies toApache Kafka 4.x; confluent-kafka Python client 2.x

Test at the Right Boundaries

Use a layered strategy rather than one test type.

Test level Kafka dependency What it validates
Unit Mocked producer/consumer Business logic, retries, serialization
Component Real ephemeral cluster Configuration, partitions, groups, commits, transactions
Contract Producer/consumer schemas Payload compatibility, event evolution
End-to-end Connect, Streams, Registry Workflows and interoperability
Failure/chaos Brokers stopped or delayed Recovery, retries, duplicates, dead-letter behaviour

Do not mock the Kafka protocol in component tests: mocks hide serialization, metadata refresh, rebalance, transaction and offset-management bugs.

Build an Isolated Test Environment

  • Run an ephemeral Kafka environment in CI with Testcontainers, Docker or an IaC-managed sandbox.

  • Kafka 4.x is KRaft-only: ZooKeeper metadata mode was removed in Kafka 4.0.

  • Give every test run a unique topic and consumer group to avoid offset and data contamination.

  • For production-like clusters use replication factor 3, min.insync.replicas=2 and unclean.leader.election.enable=false.

  • Set offsets.topic.replication.factor, transaction.state.log.replication.factor and transaction.state.log.min.isr when testing transactions.

  • Exercise Kafka Connect worker config, converters, error handling and connector plugins against the same broker.

  • Use TopologyTestDriver for fast Kafka Streams topology checks, then run the real Streams runtime in integration tests.

  • Validate Schema Registry compatibility modes and put schema evolution into contract tests.

Cover Delivery, Failure and Data Contracts

  • Ordering holds only within a partition, never across a topic; use a consistent partition key when order matters.

  • At-least-once delivery: acknowledge the broker and commit offsets only after successful processing; a crash between processing and commit yields duplicates.

  • Exactly-once semantics need idempotent producers, Kafka transactions and isolation.level=read_committed; they do not cover external databases or HTTP side effects, which need an outbox or idempotency key.

  • Test unavailable brokers, broker restarts, leader elections, rebalances, delayed responses, disk pressure and malformed events.

  • Verify retry limits, backoff, circuit breaking, poison-message handling and dead-letter topics.

  • Use unique records and explicit timeouts so a failed test cannot consume another test’s messages.

Code Example

A pytest fixture creates an isolated topic and checks a producer-consumer round trip; in application tests, call your Kafka adapter instead of produce/consume directly.

import os
import time
import uuid

import pytest
from confluent_kafka import Consumer, Producer
from confluent_kafka.admin import AdminClient, NewTopic

BOOTSTRAP = os.getenv("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092")


@pytest.fixture
def kafka_topic():
    topic = f"orders-test-{uuid.uuid4().hex}"
    admin = AdminClient({"bootstrap.servers": BOOTSTRAP})

    future = admin.create_topics([
        NewTopic(
            topic,
            num_partitions=3,
            # RF=1 because the ephemeral test cluster is a single broker.
            replication_factor=1,
            config={"cleanup.policy": "delete"},
        )
    ])[topic]
    future.result(timeout=10)

    yield topic

    try:
        admin.delete_topics([topic])[topic].result(timeout=10)
    except Exception:
        pass  # Cleanup must not hide the original test failure.


def test_order_round_trip(kafka_topic):
    producer = Producer({"bootstrap.servers": BOOTSTRAP})
    delivery_errors = []

    def on_delivery(error, message):
        delivery_errors.append(error)

    producer.produce(
        kafka_topic,
        key=b"order-1",
        value=b'{"id": 42}',
        on_delivery=on_delivery,
    )
    assert producer.flush(10) == 0
    assert len(delivery_errors) == 1
    assert delivery_errors[0] is None

    consumer = Consumer({
        "bootstrap.servers": BOOTSTRAP,
        "group.id": f"test-{uuid.uuid4().hex}",
        "auto.offset.reset": "earliest",
        "enable.auto.commit": False,
    })

    try:
        consumer.subscribe([kafka_topic])
        deadline = time.monotonic() + 10
        received = None

        while time.monotonic() < deadline:
            message = consumer.poll(1.0)
            if message is None:
                continue
            if message.error():
                raise RuntimeError(message.error())
            received = message
            break

        assert received is not None
        assert received.key() == b"order-1"
        assert received.value() == b'{"id": 42}'
    finally:
        consumer.close()

A CLI smoke test checks broker and topic operations independently:

bin/kafka-topics.sh --bootstrap-server localhost:9092 \
  --create --topic orders-smoke --partitions 3 --replication-factor 1

printf '{"id":42}\n' |
  bin/kafka-console-producer.sh \
    --bootstrap-server localhost:9092 \
    --topic orders-smoke

bin/kafka-console-consumer.sh \
  --bootstrap-server localhost:9092 \
  --topic orders-smoke \
  --group orders-smoke-group \
  --from-beginning \
  --max-messages 1

Production and CI Practices

  • Treat the test cluster as untrusted: use TLS and SASL, least-privilege ACLs and per-environment credentials and topics.

  • Monitor consumer lag, under-replicated and offline partitions, request latency, broker disk usage and failed connector tasks.

  • Make tests deterministic with bounded timeouts, controlled fixtures and assertions on success and failure paths.

  • Test rolling upgrades and mixed-version compatibility before upgrading production brokers.

  • Replication is not a backup: verify mirrored clusters and restore procedures in separate DR drills.

  • Keep production brokers out of CI; use ephemeral or dedicated non-production clusters with production-like partition counts, replication, security and monitoring.

72

What are common Apache Kafka anti-patterns?

Kafka anti-patterns are designs that look scalable or simple but cause data loss, duplicate processing, operational instability, or poor performance under production load. Avoiding them means matching partitioning, replication, consumer semantics, retention and monitoring to the system’s real failure model.

Read full answer

Applies toApache Kafka 4.x (KRaft only — ZooKeeper was removed in 4.0)

Architecture and Topic Design

Anti-pattern Production impact Safer design
One topic per entity or tenant Metadata, ACL and operational overhead grow fast A bounded set of shared topics partitioned by a stable key
Too few partitions Consumers cannot scale; hot keys bottleneck Size partitions from throughput and concurrency needs
One-partition critical topic No parallel consumption or partition failover Multiple partitions with replication factor 3
Oversized or tiny records Large records pressure memory and network; tiny ones waste overhead Keep records moderate and batch where useful
Keyless or random partitioning Related events can be reordered A stable key routes related records to one partition
  • Hot partition: a popular key (one customer) sends all traffic to one partition, capping consumer parallelism and making that partition a single point of failure.

  • Changing partition counts carelessly: adding partitions re-maps keyed records, so stateful consumers can see incompatible assignments; settle the key strategy first.

  • Replication treated as backup: replicas protect availability, not logic errors; a bad producer’s records replicate everywhere, so keep independent backups.

  • Unsafe quorum settings: replication factor 3 with min.insync.replicas=2 tolerates one broker failure and rejects writes lacking quorum; leave unclean.leader.election.enable=false unless forced availability matters more.

  • Fragile KRaft deployments: run an odd-sized controller quorum (normally three controllers) across failure domains, and separate controller and broker resources where availability demands it.

Producers, Consumers, and Guarantees

  • Non-idempotent producers: a retried produce after a network failure can duplicate records; set enable.idempotence=true with acks=all and a nonzero retries, remembering idempotence covers the producer session, not application resubmits.

  • Wrong ordering assumptions: ordering is guaranteed per partition only; same-key records must land in the same partition and be consumed in order.

  • Committing offsets before processing: this loses events permanently; with enable.auto.commit=false, commit offsets only after successful processing (at-least-once).

  • Processing inside the poll loop: long work blocks poll() and triggers max.poll.interval.ms eviction and rebalance storms; poll often and use a bounded worker pool.

  • Ignoring rebalances: deployments, partition changes and slow consumers cause churn; use cooperative sticky assignment and keep processing bounded.

  • Assuming exactly-once everywhere: exactly-once needs a Kafka transaction (sendOffsetsToTransaction in the Java client, send_offsets_to_transaction in confluent-kafka) plus isolation.level=read_committed consumers; external databases and APIs remain at-least-once unless they join the transaction.

  • Unbounded retries or timeouts: tune delivery.timeout.ms and request.timeout.ms so broker failure surfaces before the application’s own deadline.

Connect, Streams, and Schemas

  • Excessive Connect concurrency: a high tasks.max can overload the sink, exhaust connection limits and add transactional overhead; start conservative and scale from measured sink capacity.

  • Assuming every connector is exactly-once: check the connector’s delivery guarantee and the sink’s idempotence; repeated writes are otherwise possible.

  • Unbounded Streams state: stateful operators can exhaust local disk and memory; watch state-store size, changelog lag and restore time, and repartition when key semantics change.

  • Ignoring Streams transactions: processing.guarantee=exactly_once_v2 needs transactional internal topics and sufficient replication; it still excludes arbitrary external side effects.

  • Schema Registry without compatibility checks: disabling compatibility breaks consumers during record evolution; enforce a compatibility mode per environment plus a subject naming strategy.

Production Operations and Security

  • No lag or replication monitoring: watch group lag, under-replicated and offline partitions, ISR churn, request latency, broker disk usage, GC pauses, Connect task failures and Streams thread lag.

  • Alerting on lag alone: lag spikes can be normal; combine lag with processing rate, age of the oldest unprocessed record and error rate.

  • Unplanned upgrades: follow the supported upgrade path, roll brokers one at a time, and watch rebalances, ISR movement and latency after each step.

  • Plaintext or over-broad access: use TLS for client and inter-broker traffic, SASL/SCRAM for authentication and topic ACLs for least privilege; protect admin credentials and audit ACL changes.

  • Missing recovery procedures: document broker replacement, controller recovery, topic restore, offset-reset decisions and data validation; back up Schema Registry separately, since topic replication does not protect it.

Guardrails

Verify the real cluster with kafka-topics.sh --describe, kafka-consumer-groups.sh --describe and kafka-configs.sh --describe instead of trusting application metrics alone, and gate topic creation, replication settings, producer retry behaviour, commit timing, schema compatibility, capacity limits, security configuration and tested recovery drills through review.

73

How do I write a Kafka producer and consumer with confluent-kafka?

confluent-kafka is Kafka’s Python client, backed by librdkafka. A producer sends serialized records to topic partitions, while a consumer polls those partitions as part of a consumer group and can commit offsets after processing for at-least-once delivery.

Read full answer

Applies toconfluent-kafka 2.x with Apache Kafka 3.x and 4.x brokers

Architecture and Client Behavior

  • Brokers: Kafka 4.x requires KRaft for controller quorum and metadata management. Kafka 2.8 introduced KRaft as an early-access preview, and Kafka 3.3 made it production-ready.

  • Partitions: Each partition is an ordered log. Partitioning allows a topic’s throughput to scale across brokers.

  • Replication: Partition leaders replicate records to followers. For important topics, use at least three brokers and a replication factor of three.

  • Consumer groups: A group assigns each partition to at most one consumer. Different groups consume the records independently.

  • Offsets: Consumer progress is stored in Kafka’s internal __consumer_offsets topic, which uses log compaction.

  • Partition selection: With no explicit partition, librdkafka uses the configured partitioner, normally deriving the partition from a non-null key. Increasing a topic’s partition count can change that mapping.

Delivery Semantics

Semantic Required behavior
At-most-once Commit before processing; a failure can lose unprocessed records.
At-least-once Process successfully before committing; a failure between those steps can cause duplicates.
Exactly-once Use a transactional producer, isolation.level=read_committed, disabled automatic offset commits, and atomically publish output and consumed offsets with send_offsets_to_transaction().

Ordering is guaranteed only within a partition. Idempotence prevents duplicates caused by producer retries, but not application-level resending after a restart. Exactly-once workflows use init_transactions(), begin_transaction(), send_offsets_to_transaction(), and commit_transaction(). A transactional producer requires a stable, unique transactional.id; conflicting instances are fenced.

Code Example

Install the client and create a topic. The replication factor below requires three brokers:

python -m pip install "confluent-kafka>=2,<3"

export KAFKA_BOOTSTRAP_SERVERS="kafka.example.com:9092"
export KAFKA_USERNAME="example-user"
export KAFKA_PASSWORD="password"

bin/kafka-topics.sh \
  --bootstrap-server "$KAFKA_BOOTSTRAP_SERVERS" \
  --create \
  --if-not-exists \
  --topic orders \
  --partitions 6 \
  --replication-factor 3

The following program contains both a producer and consumer:

# kafka_app.py
import argparse
import json
import os

from confluent_kafka import Consumer, KafkaError, KafkaException, Producer


def make_config(client_id, **extra):
    config = {
        "bootstrap.servers": os.environ["KAFKA_BOOTSTRAP_SERVERS"],
        "security.protocol": os.getenv("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT"),
        "client.id": client_id,
    }
    config.update(extra)

    if config["security.protocol"].startswith("SASL"):
        config.update(
            {
                "sasl.mechanism": os.getenv(
                    "KAFKA_SASL_MECHANISM", "SCRAM-SHA-512"
                ),
                "sasl.username": os.environ["KAFKA_USERNAME"],
                "sasl.password": os.environ["KAFKA_PASSWORD"],
            }
        )

    ca_location = os.getenv("KAFKA_SSL_CA_LOCATION")
    if ca_location:
        config["ssl.ca.location"] = ca_location

    return config


def produce():
    failures = []

    def on_delivery(error, message):
        if error is not None:
            failures.append(str(error))
            return

        key = message.key()
        print(
            f"Delivered {key.decode() if key else '<null>'} to "
            f"{message.topic()}[{message.partition()}] "
            f"at offset {message.offset()}"
        )

    producer = Producer(
        make_config(
            "orders-producer",
            **{
                "acks": "all",
                "enable.idempotence": True,
                "linger.ms": 5,
            },
        )
    )

    try:
        for order_id in ("order-1001", "order-1002", "order-1003"):
            event = {"order_id": order_id, "status": "created"}
            key = b"customer-42"

            while True:
                try:
                    producer.produce(
                        topic="orders",
                        key=key,
                        value=json.dumps(event).encode("utf-8"),
                        on_delivery=on_delivery,
                    )
                    break
                except BufferError:
                    # Serve delivery callbacks while the local queue is full.
                    producer.poll(1)

            producer.poll(0)
    finally:
        remaining = producer.flush(30)
        if remaining:
            raise TimeoutError(f"{remaining} messages were not delivered")
        if failures:
            raise RuntimeError(f"Delivery failed: {failures}")


def consume():
    consumer = Consumer(
        make_config(
            "orders-consumer",
            **{
                "group.id": "orders-processors",
                "enable.auto.commit": False,
                "auto.offset.reset": "earliest",
                "isolation.level": "read_committed",
            },
        )
    )

    try:
        consumer.subscribe(["orders"])

        while True:
            message = consumer.poll(1.0)
            if message is None:
                continue

            if message.error():
                if message.error().code() == KafkaError._PARTITION_EOF:
                    continue
                raise KafkaException(message.error())

            event = json.loads(message.value().decode("utf-8"))
            print(f"Processing {event}")

            # Commit only after the business side effect succeeds.
            consumer.commit(message=message, asynchronous=False)
    except KeyboardInterrupt:
        pass
    finally:
        consumer.close()


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("mode", choices=("produce", "consume"))
    args = parser.parse_args()

    if args.mode == "produce":
        produce()
    else:
        consume()

Run each role separately:

python kafka_app.py produce
python kafka_app.py consume

For a secured cluster, set KAFKA_SECURITY_PROTOCOL=SASL_SSL, provide the SASL environment variables, and add --command-config /path/to/client.properties to the administrative topic command.

Production Operations

  • Monitoring: Track consumer lag, under-replicated and offline partitions, ISR changes, request latency, disk usage, producer errors, and rebalance frequency.

  • Retention: Configure retention.ms or retention.bytes. Use cleanup.policy=compact for state-like topics that retain the latest value per key.

  • Disaster recovery: Replication is not a backup. Use cross-region replication or mirroring and regularly test restoration procedures.

  • Ecosystem: Kafka Connect runs source and sink connectors; Kafka Streams provides stateful stream processing; Schema Registry manages versioned Avro, JSON Schema, and Protobuf schemas.

  • Security and upgrades: Use TLS or mTLS, SASL, least-privilege ACLs, and secret-manager injection. Upgrade brokers in a supported rolling sequence before upgrading clients.

Best Practices and Common Pitfalls

  • Use acks=all and idempotence for retry-safe, durable production.

  • Treat processing and offset commit as separate steps; duplicates are preferable to data loss.

  • Route malformed or repeatedly failing records to a dead-letter topic.

  • Avoid excessive partition counts and use snappy or zstd compression where supported.

  • Do not commit credentials; inject the SASL password through the runtime secret store.

74

How to create Kafka topics and inspect consumer groups with CLI commands

Use Apache Kafka CLI tools to create and describe topics with kafka-topics.sh, then inspect consumer-group membership, committed offsets, and lag with kafka-consumer-groups.sh. The commands connect through --bootstrap-server, so they do not require direct access to every broker.

Read full answer

Applies toApache Kafka 4.x in KRaft mode

Code Example

Run these commands from the Kafka installation directory:

# Create a topic with six partitions and three replicas.
bin/kafka-topics.sh \
  --bootstrap-server broker-1:9092 \
  --create \
  --if-not-exists \
  --topic orders \
  --partitions 6 \
  --replication-factor 3 \
  --config min.insync.replicas=2

# List topics.
bin/kafka-topics.sh \
  --bootstrap-server broker-1:9092 \
  --list

# Show partition leaders, replicas, in-sync replicas, and topic configuration.
bin/kafka-topics.sh \
  --bootstrap-server broker-1:9092 \
  --describe \
  --topic orders

# List consumer groups known to the cluster.
bin/kafka-consumer-groups.sh \
  --bootstrap-server broker-1:9092 \
  --list

# Show members, partition assignments, committed offsets, and lag.
bin/kafka-consumer-groups.sh \
  --bootstrap-server broker-1:9092 \
  --describe \
  --group order-processor

If the cluster requires TLS or SASL authentication, pass a client properties file to both tools:

bin/kafka-consumer-groups.sh \
  --bootstrap-server broker-1:9092 \
  --command-config /etc/kafka/client.properties \
  --describe \
  --group order-processor

The properties file should contain the appropriate security.protocol, SASL, and SSL settings.

What the Inspection Shows

Command Important output
kafka-topics.sh --list Names of visible topics
kafka-topics.sh --describe --topic orders Partition count, replication factor, leader, replicas, ISR, and configuration
kafka-consumer-groups.sh --list Consumer-group IDs
kafka-consumer-groups.sh --describe --group order-processor Group state, active members, assignments, current offset, log-end offset, and lag

Consumer-group lag is approximately:

lag = log-end offset - current committed offset

It represents records not yet consumed by the group, although short-lived differences are normal while consumers process records or commits are delayed.

How It Works

  • Topic creation: kafka-topics.sh sends a create request through a broker to the active KRaft controller. The controller creates the partition metadata and assigns replica leaders.

  • Replication: Each partition has one leader and configured follower replicas. Only leaders serve reads and writes for that partition.

  • In-sync replicas: Replicas that have caught up appear in the ISR. With min.insync.replicas=2, an acks=all producer succeeds only while at least two replicas are in sync.

  • Consumer groups: Kafka’s group coordinator assigns partitions among group members. Committed offsets are stored in the internal __consumer_offsets topic; do not modify that topic directly.

  • Ordering: Kafka preserves record order within a partition, not across an entire topic. A consumer group normally assigns each partition to only one member at a time.

Guarantees and Caveats

  • CLI inspection does not change producer or consumer delivery semantics.

  • For reliable at-least-once processing, disable automatic offset commits and commit the offset only after successful processing. A crash before the commit can cause reprocessing and duplicates.

  • Kafka’s transactional read-process-write flow provides exactly-once behavior within Kafka. It requires a transactional producer with transactional.id, enable.idempotence=true, and acks=all; the consumer must use isolation.level=read_committed. Offsets and output records must be included in the same transaction.

  • Kafka Streams applications can use processing.guarantee=exactly_once, subject to correctly configured transaction-state replication.

  • Exactly-once does not make non-transactional databases, HTTP calls, or other external side effects atomic.

Production Practices

  • Use replication factor 3 across separate failure domains and set min.insync.replicas=2 for important topics. Creation fails if the cluster cannot satisfy the requested replication factor.

  • Monitor consumer lag, under-replicated partitions, offline partitions, request latency, disk utilization, and broker failures. Sustained lag is usually a capacity, processing-time, or rebalance problem.

  • Expect temporary command failures during controller elections, broker outages, coordinator transitions, or network partitions; retry with bounded backoff and alerts.

  • Grant least-privilege ACLs for topic creation, topic description, group description, and cluster metadata. Administrative operations should use encrypted connections and protected client configuration files.

75

How do you design an event-driven order-processing system with Kafka?

An event-driven order-processing system uses Kafka as the durable event backbone: services publish immutable events, partition them by orderId, and consume them through groups with idempotent business processing. This provides at-least-once transport while avoiding duplicate business effects.

Read full answer

Applies toApache Kafka 4.x (KRaft); Python 3.12 with confluent-kafka 2.x; Confluent Schema Registry 8.x

Architecture

  • Controllers: Run three KRaft controllers across failure domains. Kafka 4.x does not use ZooKeeper.

  • Brokers and partitions: Each topic has partitions, each with one leader and replicas. Start with replication factor 3 and min.insync.replicas=2 across availability zones.

  • Producers: Set the record key to orderId, preserving ordering only for that order. Include eventId, schema version, event type, timestamp, and payload.

  • Consumer groups: Give each independently scalable service its own group, such as order-processors or notifications. A group assigns each partition to one member; committed offsets are stored in the internal __consumer_offsets topic by default.

  • Database bridge: Write the business row and an outbox row in one database transaction. Debezium on Kafka Connect can capture the outbox change into Kafka, avoiding an atomic database-to-Kafka commit.

Delivery Guarantees

Semantics Configuration and behavior Trade-off
At most once Use producer acks=0, or commit a consumer offset before processing Fastest, but events can be lost
At least once Use acks=all, enable.idempotence=true, and commit offsets after processing Recommended; retries or crashes can cause duplicates
Exactly once within Kafka Use a transactional producer with a unique, stable transactional.id, include source offsets in the transaction, and consume with isolation.level=read_committed Kafka read-process-write is atomic; external systems are not

Idempotent producers suppress retries caused by producer failures, but not intentional or cross-session duplicate events. Apply the inbox pattern: enforce a unique eventId and perform the inbox insert and order-state update in the same database transaction. This provides effectively-once business effects over at-least-once delivery.

Data Management

  • Retention: Use cleanup.policy=delete for event history. A seven-day retention period is illustrative; regulated records may require longer retention or archival.

  • Compaction: Use a separate compacted order-state topic keyed by orderId. Compaction is eventual; tombstones mark keys as deleted and remain for delete.retention.ms.

  • Schema Registry: Register Avro, JSON Schema, or Protobuf schemas in Schema Registry rather than embedding schemas in every event. Schema Registry persists schemas in its internal _schemas topic by default; use BACKWARD compatibility unless another migration policy is required.

  • Kafka Connect: Use Debezium for CDC and connector-specific sinks for projections. Source exactly-once support requires workers with exactly.once.source.support=enabled and a capable connector (exactly.once.support=required); sink support is connector-specific, so otherwise use idempotent sinks.

  • Kafka Streams: Use it for stateful aggregation within the event pipeline. Set processing.guarantee=exactly_once_v2 when its Kafka read-process-write transaction model is required.

Operations and Failure Modes

  • Monitor consumer lag, rebalance frequency, produce and consume latency, error rates, disk usage, under-minimum-ISR partitions, offline partitions, leader elections, and broker quotas.

  • With acks=all and min.insync.replicas=2, writes become unavailable if the in-sync replica count falls below two. A broker failure normally causes leader election and continued service when sufficient replicas remain.

  • Poison messages can block a partition. Move repeatedly failing events to bounded retry topics and then a dead-letter topic, preserving the original key and adding error metadata.

  • Keep processing within max.poll.interval.ms; reduce batch work or use bounded batches rather than merely increasing the timeout. Monitor hot orderId keys and partition traffic.

  • Use TLS for clients and inter-broker traffic, SASL/SCRAM or OAUTHBEARER authentication, least-privilege ACLs, and no unauthenticated production listener.

  • Replication is not a backup. Use MirrorMaker 2 for another region, archive important events to durable object storage, and test restoration procedures.

Code Example

Create event and compacted-state topics with the required ISR threshold:

kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092 \
  --create --if-not-exists --topic order-events \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000 \
  --config min.insync.replicas=2

kafka-topics.sh \
  --bootstrap-server kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092 \
  --create --if-not-exists --topic order-state \
  --partitions 12 --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.insync.replicas=2

The following consumer demonstrates inbox deduplication before committing its offset. In production, put the order-state mutation in the same database transaction as the inbox insert.

import json
import sqlite3
from contextlib import closing

from confluent_kafka import Consumer, KafkaError

DB = "orders.db"
BOOTSTRAP = "kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092"

with closing(sqlite3.connect(DB)) as db, db:
    with closing(db.execute(
        "CREATE TABLE IF NOT EXISTS processed_events "
        "(event_id TEXT PRIMARY KEY NOT NULL, payload TEXT NOT NULL)"
    )):
        pass

def persist_once(raw):
    event = json.loads(raw)
    with closing(sqlite3.connect(DB)) as db, db:
        with closing(db.execute(
            "INSERT INTO processed_events(event_id, payload) VALUES (?, ?) "
            "ON CONFLICT(event_id) DO NOTHING",
            (event["eventId"], raw.decode("utf-8")),
        )):
            pass

consumer = Consumer({
    "bootstrap.servers": BOOTSTRAP,
    "group.id": "order-processors",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",
})

try:
    consumer.subscribe(["order-events"])
    while True:
        message = consumer.poll(1.0)
        if message is None:
            continue
        if message.error():
            if message.error().code() == KafkaError._PARTITION_EOF:
                continue
            raise RuntimeError(str(message.error()))

        persist_once(message.value())
        consumer.commit(message=message, asynchronous=False)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Production Best Practices

  • Test broker, network, controller, disk, consumer, rebalance, and replay scenarios.

  • Exercise schema upgrades and partition expansion before production rollout.

  • Keep sensitive payloads out of logs, encrypt stored data where required, and never reuse an eventId for different business actions.

76

How can Kafka guarantee exactly-once payment event processing?

Exactly-once processing requires Kafka offsets, internal state, and produced records to be committed atomically. Kafka can provide this guarantee for Kafka-to-Kafka processing, but an external payment or database operation also requires an idempotency key or transactional-outbox pattern because Kafka cannot atomically commit an external system.

Read full answer

Applies toApache Kafka 4.x with Kafka Streams and KRaft

Guarantee Boundary

Scenario Guarantee
At-least-once consumer The handler may run again after a crash, causing duplicate effects
Kafka transactional processing State, offsets, and Kafka output commit atomically, producing one visible output per accepted input record
External payment or database Not covered by Kafka transactions; the external operation must be idempotent

Exactly-once also does not mean application code executes only once. A Kafka Streams processor can rerun after an aborted transaction or crash. The committed Kafka result is exactly once, while the business operation must tolerate retries safely.

Required Architecture

  • Use Kafka Streams with processing.guarantee=exactly_once_v2, or manually coordinate a transactional producer with consumer offsets.

  • Enable producer idempotence with enable.idempotence=true, acks=all, and no more than five in-flight requests per connection.

  • Use a stable transactional.id for manually transactional producers. Kafka Streams manages transactional IDs using its application.id.

  • Commit the consumed offset, state-store change, and output record in the same transaction.

  • Configure downstream consumers with isolation.level=read_committed; otherwise they can observe records from aborted transactions.

  • Use a stable payment_id as the partition key so all events for one payment stay in order, and carry a unique event_id in each record for deduplication. Ordering holds per partition, not globally across the topic.

  • Ensure each payment event ID is unique upstream. Idempotent production prevents retry duplicates, while topic compaction only controls long-term log size and does not provide immediate deduplication.

How It Works

With exactly_once_v2, Kafka Streams runs one transaction per stream thread, covering all of that thread’s tasks. It writes output records and updated state to internal changelog topics, then commits those records together with the consumed source offsets. If the process fails before commit, Kafka aborts the transaction and reprocesses the input after restoration.

Consumers using read_committed see only committed transactional records. Ordering is guaranteed within each partition; Kafka does not guarantee global ordering across partitions.

For an external payment provider, retry the same operation with the event_id as the provider’s idempotency key. For a database, enforce a unique constraint on event_id and update the payment record and an outbox row in one database transaction. A distributed transaction between Kafka, a database, and a payment provider generally does not exist.

Code Example

Create replicated input and output topics across three brokers. Event topics use delete; compacting them by key would discard earlier events of a payment:

for topic in payments payments-processed; do
  bin/kafka-topics.sh \
    --bootstrap-server kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092 \
    --create \
    --topic "$topic" \
    --partitions 12 \
    --replication-factor 3 \
    --config cleanup.policy=delete
done

Use these broker settings in server.properties:

min.insync.replicas=2
unclean.leader.election.enable=false
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=2

A minimal Kafka Streams application using the Kafka Streams 4.x API:

import java.util.Properties;
import java.util.concurrent.CountDownLatch;

import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.KafkaStreams;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.kstream.Consumed;

public final class PaymentProcessor {
    public static void main(String[] args) throws InterruptedException {
        Properties config = new Properties();
        config.put(StreamsConfig.APPLICATION_ID_CONFIG, "payment-processor-v1");
        config.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka.example.com:9092");
        config.put(
                StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,
                Serdes.StringSerde.class);
        config.put(
                StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG,
                Serdes.StringSerde.class);
        config.put(
                StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
                StreamsConfig.EXACTLY_ONCE_V2);
        config.put(StreamsConfig.STATE_DIR_CONFIG, "/var/lib/kafka-streams");

        StreamsBuilder builder = new StreamsBuilder();
        builder
                .stream("payments", Consumed.with(Serdes.String(), Serdes.String()))
                .mapValues(value -> "processed:" + value)
                .to("payments-processed");

        KafkaStreams streams = new KafkaStreams(builder.build(config), config);
        CountDownLatch stopped = new CountDownLatch(1);

        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            streams.close();
            stopped.countDown();
        }));

        streams.start();
        stopped.await();
    }
}

Replace the deterministic mapValues operation with the payment business logic. If that logic calls a provider or database, implement the separate idempotency mechanism described above.

Operations and Failure Handling

  • Monitor consumer lag, transaction aborts, transaction timeouts, under-replicated partitions, producer retry rates, and offline replicas.

  • Inspect lag with:

bin/kafka-consumer-groups.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe \
  --group payment-processor-v1
  • A transaction timeout aborts work and causes reprocessing. Keep business operations fast and configure timeouts consistently across producer, broker, and application.

  • Kafka Streams state is restored from changelog topics. Test restoration, configure rack awareness, and use standby replicas where availability requirements justify them.

  • Perform rolling upgrades only with compatible clients and verify that transactional processing remains enabled.

  • Protect brokers with TLS and SASL, and restrict topic, consumer-group, and transactional-topic ACLs.

Best Practices and Pitfalls

  • Treat Kafka exactly-once as an end-to-end design, not a single broker setting.

  • Kafka Connect provides exactly-once only for connectors that explicitly support transactional source offsets and sinks. Otherwise, use at-least-once plus a unique external key.

  • Schema Registry can enforce payment-event schemas and compatibility, but it does not provide exactly-once delivery by itself.

  • Do not assume global event ordering; partition by a stable key such as payment_id.

  • Keep the payment event ID stable across every retry, state update, outbox record, and external API call.

77

How do you troubleshoot rapidly growing Kafka consumer group lag?

Consumer lag is the difference between a partition’s log-end offset and the offset committed by its consumer group. Rapidly increasing lag usually means production exceeds the group’s effective processing capacity, although stopped consumers, rebalances, broker failures, or downstream outages can cause the same symptom.

Read full answer

Applies toApache Kafka 3.6-4.x

Diagnose the Bottleneck

Kafka stores each topic in partitions whose logs are replicated across brokers; one partition is assigned to only one member of a consumer group.

  • Committed versus live position: --describe reports the committed offset. With auto-commit enabled, it can overstate the actual backlog by up to the commit interval; correlate it with consumer metrics such as records-lag-max and the processed-record rate.

  • Active members and assignment: Check group state, member count, and lag per partition. A group with no active consumers cannot make progress, and a hot partition limits parallelism within that group.

  • Rate comparison: Compare producer and consumer rates; a sustained difference explains continued growth.

  • Rebalance and downstream health: Frequent rebalances interrupt polling; database calls, HTTP requests, locks, CPU saturation, or a downstream outage can make consumers appear healthy while processing very slowly.

  • Broker and metadata health: Check unavailable or under-replicated partitions, disk, network, request latency, and send latency. Kafka 3.6 supports KRaft or ZooKeeper; Kafka 4.x is KRaft-only, so also inspect controller-quorum health where applicable.

  • Other client types: For Kafka Connect, inspect connector and task status plus task lag; for Kafka Streams, inspect thread/client lag, processing and restore latency, and state-store performance; for Schema Registry, inspect compatibility or validation errors that cause records to be rejected or retried as poison records.

Offset lag is an offset delta, not necessarily the number of retained records: compaction and control or aborted transaction records can leave offset gaps. Estimate backlog age from record timestamps and use it alongside lag.

Safely Reduce the Backlog

  1. Scale consumers when partitions and downstream capacity allow it. Add members up to the number of partitions assigned to the topic; additional consumers remain idle.

  2. Fix hot partitions if concurrency is limited. Increasing partition count enables parallelism, but it can change the key-to-partition mapping, so validate ordering requirements first.

  3. Reduce processing time by batching downstream writes, removing unnecessary remote calls, caching stable lookups, or using asynchronous processing with controlled concurrency.

  4. Tune polling and fetching carefully. Increasing max.poll.records reduces poll overhead but increases batch size and the chance of exceeding max.poll.interval.ms.

  5. Reduce rebalance impact with CooperativeStickyAssignor and, where appropriate, static membership through a unique, stable group.instance.id.

  6. Control upstream traffic during an incident if necessary and with business approval. This slows backlog growth and buys drain time, but it does not extend retention or prevent existing records from expiring.

Do not blindly increase max.poll.interval.ms; it allows slow processing to continue but does not improve throughput and can delay failure detection.

Preserve Delivery and Ordering Guarantees

Kafka provides ordering within a partition, not across partitions. Consumer-group membership does not change that boundary.

  • At-least-once: Disable automatic offset commits, process the records successfully, and then commit their offsets. A crash before the commit causes reprocessing but avoids data loss.

  • Exactly-once: For a consume-transform-produce workflow, use an idempotent transactional producer with a stable, unique transactional.id per producer instance, send consumed offsets to the same transaction with sendOffsetsToTransaction, and consume with isolation.level=read_committed.

  • Read-committed consumers: They stop at the last stable offset when a producer transaction remains open. An unusually long-running transaction can therefore look like lag.

  • Retries: Bound retries and isolate poison messages. Infinite retries inside the poll loop can trigger consumer eviction.

Exactly-once processing adds transaction-coordination overhead and does not make inherently slow application logic faster. External database or HTTP side effects still require idempotency or an outbox.

Code Example

Inspect the group, partition layout, and broker replication state:

kafka-consumer-groups.sh \
  --bootstrap-server kafka.example.com:9092 \
  --group order-processors \
  --describe

kafka-consumer-groups.sh \
  --bootstrap-server kafka.example.com:9092 \
  --group order-processors \
  --describe --members --verbose

kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe --topic orders

kafka-topics.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe --under-replicated-partitions --unavailable-partitions

For secured clusters, pass --command-config client.properties containing the SASL/TLS settings.

Example consumer tuning, used only after measuring actual processing time:

max.poll.records=1000
max.poll.interval.ms=300000
partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor
enable.auto.commit=false

For a static consumer, also configure a unique identifier:

group.instance.id=order-processor-1

Production Practices and Pitfalls

  • Monitor partition lag, backlog age, processing rate, rebalance frequency, commit failures, broker request latency, under-replication, disk usage, and producer backpressure.

  • Alert on trends rather than a single offset threshold; a growing backlog is usually more actionable than a temporarily high value.

  • Check delete-retention deadlines and disk headroom. Monitoring alone cannot extend retention, so use an approved retention change or recovery plan before data expiry.

  • Use replication and tested cluster-level backups; consumer offsets are not a substitute for data backups.

  • Prefer TLS and SASL authentication, and grant only required topic and group ACL permissions.

  • Before rolling upgrades, confirm compatibility, healthy replication, sufficient capacity, and a plan for draining or reassigning consumer groups.

  • Avoid resetting offsets to the latest during an incident unless data loss is explicitly accepted; that hides the backlog rather than resolving it.

78

How do you migrate relational database polling to Kafka events?

Use a transactional outbox to convert committed relational changes into Kafka events without an unsafe database-to-Kafka dual write. Migrate with parallel shadow validation, then switch consumers and retire polling only after parity, reliability, and rollback criteria are met.

Read full answer

Applies toApache Kafka 4.x (KRaft), confluent-kafka Python 2.x, PostgreSQL 14+

Migration Architecture

  • Write the business change and an event_outbox row in the same database transaction.

  • In a bounded relay transaction, select unpublished rows with FOR UPDATE SKIP LOCKED, publish them to Kafka, wait for acknowledgement, set published_at, and commit the transaction. Roll back on failure so another relay can retry.

  • Give each event an immutable event_id, schema_version, event type, and aggregate key. Use the aggregate ID as the Kafka key to preserve ordering for that entity.

  • Partition a topic across multiple brokers. For example, use 12 partitions, replication factor 3, and min.insync.replicas=2.

  • Kafka 4.x uses KRaft: controllers maintain cluster metadata, while brokers store and replicate partition data.

  • For high-volume or low-latency change capture, a Kafka Connect source such as Debezium can replace the custom relay when every captured change is mapped into the required event contract.

  • Use Kafka Connect for sink connectors, Kafka Streams for stateful transformations, and Schema Registry for versioned Avro or JSON Schema payloads.

Cutover Process

  1. Record the current poller’s correctness, freshness, throughput, and recovery requirements.

  2. Create the Kafka topic, ACLs, schema subjects, and consumer groups.

  3. Deploy the outbox and relay while continuing to run the existing polling workflow.

  4. Run shadow consumers against Kafka, compare their results with the polling path, and suppress duplicate production side effects.

  5. Switch production consumers to Kafka, monitor lag and discrepancies, and stop the poller after the rollback window. Retain the outbox temporarily for recovery and audit.

Delivery and Ordering Guarantees

  • Kafka preserves ordering within a partition, not across the topic. Using one aggregate ID as the key maps that aggregate to one partition.

  • Enable producer idempotence with acks=all, enable.idempotence=true, and max.in.flight.requests.per.connection=5.

  • Idempotence suppresses duplicates caused by retrying the same producer sequence. It does not prevent a second application-level publish, such as an outbox relay retrying after Kafka acknowledged the event but before its database transaction committed. Consumers must deduplicate using event_id, commonly with a database inbox table and unique constraint.

  • For at-least-once consumption, set enable.auto.commit=false and commit the next partition offset only after processing the event and committing its database side effect. A crash between those commits causes a duplicate rather than a loss.

  • Kafka exactly-once semantics require transactional producers and consumers using isolation.level=read_committed, with consumed offsets included in the producer transaction. They do not automatically make an arbitrary external database write exactly once; use inbox/outbox coordination or an equivalent mechanism.

Code Example

Create the outbox table:

CREATE TABLE event_outbox (
    event_id uuid PRIMARY KEY,
    schema_version smallint NOT NULL,
    topic text NOT NULL,
    event_key text NOT NULL,
    event_type text NOT NULL,
    payload jsonb NOT NULL,
    created_at timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz
);

CREATE INDEX event_outbox_unpublished_idx
    ON event_outbox (created_at)
    WHERE published_at IS NULL;

Create the topic with client properties supplied through --command-config:

bin/kafka-topics.sh --bootstrap-server kafka.example.com:9093 \
  --command-config /etc/kafka/client.properties \
  --create --topic orders.v1 --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000

Publish an outbox row with the Python client:

import json
import os
from confluent_kafka import Producer

producer = Producer({
    "bootstrap.servers": "kafka.example.com:9093",
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": "outbox-relay",
    "sasl.password": os.environ["KAFKA_PASSWORD"],
    "enable.idempotence": True,
    "acks": "all",
    "max.in.flight.requests.per.connection": 5,
    "delivery.timeout.ms": 120000,
})

def publish_outbox_row(row):
    errors = []

    def delivered(err, message):
        if err is not None:
            errors.append(str(err))

    value = json.dumps({
        "event_id": str(row["event_id"]),
        "schema_version": row["schema_version"],
        "event_type": row["event_type"],
        "payload": row["payload"],
    }).encode("utf-8")

    producer.produce(
        row["topic"],
        key=row["event_key"].encode("utf-8"),
        value=value,
        headers=[("event_type", row["event_type"])],
        on_delivery=delivered,
    )

    if producer.flush(130) != 0 or errors:
        raise RuntimeError("Kafka delivery did not complete")

publish_outbox_row({
    "event_id": "2fd33344-45d3-4c26-b5cf-8e724fe73122",
    "schema_version": 1,
    "topic": "orders.v1",
    "event_key": "order-42",
    "event_type": "OrderPlaced",
    "payload": {"order_id": "order-42", "status": "paid"},
})

Production Operations

  • Monitor consumer lag and lag age, under-replicated and offline partitions, broker disk usage, produce failures, retry volume, and dead-letter queue growth.

  • Configure bounded consumer retries and send repeatedly failing events to a versioned DLQ topic.

  • Use cleanup.policy=delete with an appropriate retention.ms for immutable event history. Use log compaction only for latest-state or changelog topics.

  • Enforce TLS and SASL authentication, least-privilege topic and group ACLs, Schema Registry authorization, and secrets management. Provision KAFKA_PASSWORD at runtime from a secret manager; never commit credentials.

  • Replication is not a backup. Use MirrorMaker 2 or another tested cross-region replication design, define recovery objectives, and rehearse restoration.

  • Upgrade brokers in a rolling manner after checking compatibility with Kafka Connect, Schema Registry, and clients. Use FORWARD or FULL compatibility when consumers may upgrade after producers; BACKWARD assumes consumers upgrade first.

79

How do you design a real-time analytics pipeline with Kafka?

A real-time analytics pipeline should use Kafka as a durable event backbone, decoupling ingestion, stream processing, and analytical sinks. Event-driven architecture provides replay, horizontal scalability, and controlled delivery semantics when topics, partitions, replication, and consumer offsets are designed explicitly.

Read full answer

Applies toApache Kafka 4.0.x in KRaft mode; confluent-kafka Python client 2.x; Confluent Schema Registry 8.x

Pipeline Architecture

  • Source producers publish events to an append-only, versioned raw topic such as orders.v1.

  • Partitions distribute events across brokers. Use a stable key such as customer_id when events for one entity must remain ordered.

  • Brokers store partition leaders and followers. With replication factor 3, a leader failure can be replaced without losing acknowledged data.

  • Consumer groups divide partitions among consumers. Each group maintains its own offsets in __consumer_offsets.

  • Stream processors validate, enrich, aggregate, and republish events. Analytical sinks such as ClickHouse, Elasticsearch, or a data warehouse receive processed results through Kafka Connect or application-specific consumers.

In KRaft mode, deploy an odd-sized controller quorum, normally three controllers across failure domains, plus multiple brokers. Controllers manage metadata; brokers handle the data path.

Topic and Retention Design

  • Set partitions according to expected throughput and required consumer parallelism; avoid excessive partitions and key-induced hotspots.

  • Use replication factor 3 and min.insync.replicas=2 for production durability.

  • Retain raw events long enough to replay and rebuild downstream state.

  • Use cleanup.policy=delete for event streams and cleanup.policy=compact for keyed state topics. Emit tombstones when compact keys must be removed.

  • Place replicas across availability zones within a cluster. Cross-region disaster recovery requires MirrorMaker 2 or another replication system; replication alone is not a backup.

Delivery Guarantees

  • At least once: Disable auto-commit, process the event, then commit its offset. A crash after processing but before committing causes redelivery, so the sink must deduplicate using an event ID or equivalent.

  • Exactly once for Kafka read-process-write: Use a transactional producer with a transactional.id that is stable across restarts and unique among concurrently active producers, enable idempotence, and configure consumers with isolation.level=read_committed. Atomically include output records and consumed offsets in the transaction using send_offsets_to_transaction.

  • Ordering: Kafka preserves ordering within a partition, not across a topic. Route related events to the same partition. Idempotent production permits up to five in-flight requests per connection while preserving order.

  • Exactly-once processing does not automatically include writes to an external database. External effects still require idempotency or a coordinated transaction.

Streaming Ecosystem

  • Kafka Streams provides stateful operators, windowed aggregation, and repartitioning. State changes are backed by changelog topics for recovery.

  • Kafka Connect runs source and sink connectors with independent scaling and offset management. Connector-specific transactional support determines whether exactly-once behavior is possible.

  • Schema Registry stores Avro, JSON Schema, or Protobuf schemas. Enforce compatibility such as BACKWARD to prevent incompatible event evolution.

Code Example

Create and inspect a production-style topic:

export KAFKA_HOME=/opt/kafka

"$KAFKA_HOME/bin/kafka-topics.sh" \
  --bootstrap-server kafka.example.com:9092 \
  --create --topic orders.v1 \
  --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config cleanup.policy=delete \
  --config retention.ms=604800000

"$KAFKA_HOME/bin/kafka-configs.sh" \
  --bootstrap-server kafka.example.com:9092 \
  --describe --entity-type topics --entity-name orders.v1

The following producer is idempotent, while the consumer implements manual offset commits:

import os
from confluent_kafka import Producer, Consumer

bootstrap = os.getenv("KAFKA_BOOTSTRAP", "kafka.example.com:9092")
security = {
    "bootstrap.servers": bootstrap,
    "security.protocol": "SASL_SSL",
    "sasl.mechanism": "SCRAM-SHA-512",
    "sasl.username": os.environ["KAFKA_USERNAME"],
    "sasl.password": os.environ["KAFKA_PASSWORD"],
}

producer = Producer({
    **security,
    "enable.idempotence": True,
    "acks": "all",
    "max.in.flight.requests.per.connection": 5,
})

delivery_errors = []

def on_delivery(error, message):
    if error is not None:
        delivery_errors.append(str(error))

producer.produce(
    "orders.v1",
    key=b"customer-42",
    value=b'{"event_id":"event-1","amount":125.50}',
    on_delivery=on_delivery,
)
producer.flush()

if delivery_errors:
    raise RuntimeError("; ".join(delivery_errors))

consumer = Consumer({
    **security,
    "group.id": "analytics",
    "enable.auto.commit": False,
    "auto.offset.reset": "earliest",
    "isolation.level": "read_committed",
})

consumer.subscribe(["orders.v1"])

try:
    while True:
        message = consumer.poll(1.0)
        if message is None:
            continue
        if message.error():
            print(f"Kafka error: {message.error()}")
            continue

        # Replace this with an idempotent, durable sink write.
        print(message.value().decode("utf-8"))
        consumer.commit(message=message, asynchronous=False)
except KeyboardInterrupt:
    pass
finally:
    consumer.close()

Production Operations

  • Monitor consumer lag by partition and lag age, offline or under-replicated partitions, broker disk usage, request latency, JVM garbage collection, and producer error rates.

  • Route repeatedly failing records to a dead-letter topic; otherwise one poison event can block a partition.

  • Use TLS for encryption in transit, SASL_SSL authentication, and least-privilege ACLs for topics, consumer groups, and Schema Registry.

  • Roll broker upgrades one at a time, checking ISR and replication health after each step. Do not lose controller quorum during the upgrade.

  • Test restoration using a separate region or MirrorMaker 2. Rebuild downstream state from retained raw events, back up schema and configuration metadata, and periodically validate backup integrity.

80

How should you choose a retention period for an Apache Kafka topic?

Choose the topic’s retention period from the replay window the business requires, plus enough time for consumers to recover from outages, while remaining within the cluster’s storage capacity. The policy must match whether the topic contains a complete event history, current state per key, or both.

Read full answer

Applies toApache Kafka 4.x

How Kafka Retention Works

For cleanup.policy=delete, Kafka removes whole log segments rather than individual records. The active segment is not deleted, so actual data lifetime can exceed the configured period until a segment rolls.

  • retention.ms: Maximum record age. The default is seven days; -1 retains data indefinitely.

  • retention.bytes: Maximum log size per partition. The default is -1, meaning no size limit.

  • Segment rolling: segment.ms and segment.bytes determine when a new segment starts. Smaller segments provide more precise cleanup but create more files.

  • Deletion timing: Retention checks and filesystem deletion are periodic rather than instantaneous. The broker-level file.delete.delay.ms controls the delay between physical file deletions.

  • Combined constraints: With cleanup.policy=delete, age or size limits can cause deletion. retention.bytes can therefore remove old segments before the nominal time window, so it is a capacity safeguard rather than a precise retention guarantee.

Choosing a Defensible Period

Base the decision on concrete recovery requirements:

  1. Replay requirement: Determine how far back an application must replay events. For example, a fraud pipeline might require 30 days for investigations.

  2. Consumer outage tolerance: Include the longest acceptable outage plus catch-up time. A consumer that falls behind by seven days may find that the required records have already expired.

  3. Recovery objective: Align the window with the business’s acceptable recovery point.

  4. Storage capacity: Estimate retained bytes as aggregate ingress bytes/second × retention seconds × replication factor, then add headroom for bursts and segment overhead. Remember that retention.bytes applies independently to each partition.

  5. Compliance: Longer retention increases cost and the amount of sensitive data retained. Apply data minimization, encryption, ACLs, and auditing.

Consumer-group offsets are retained separately through offsets.retention.minutes. This expiry begins after a consumer group becomes empty; deleting event data does not delete the corresponding offsets.

Delete vs. Compact

Policy Behavior Suitable use Important caveat
delete Removes old segments based on age or size Logs, immutable events, replayable streams Data may disappear before the exact retention.ms value
compact Retains the latest surviving record for each key State stores and CDC-style topics Older records are removed only after compaction
compact,delete Combines compaction with segment expiry Bounded state topics Both compaction and expiry behavior must be tuned

For a compacted topic, a null value is a tombstone that deletes the key during compaction. delete.retention.ms controls how long that tombstone is retained after compaction to prevent out-of-order processing from resurrecting the key. Compaction retains surviving records in partition order, but records for the same key must consistently use the same partition.

Operational and Security Considerations

  • Consumer lag: Track lag against the oldest available record, not merely the latest offset. Data can expire while a consumer still has an earlier committed offset.

  • Delivery guarantees: Ordering is guaranteed only within a partition. For at-least-once consumption, disable auto-commit and commit offsets only after processing; a crash can still repeat an external side effect. Kafka 4.x producers enable idempotence by default, but that does not make consumer side effects exactly-once.

  • Exactly-once processing: A Kafka read-process-write workflow requires idempotent production, a stable and unique transactional.id, transactional offset commits, enable.auto.commit=false, and consumers using isolation.level=read_committed. Kafka Streams applications use processing.guarantee=exactly_once_v2. External systems still require transactional integration or an outbox pattern.

  • Replication: Segment deletion is propagated across replicas asynchronously. Monitor under-replicated and offline partitions.

  • Storage: Failed cleanup can fill disks and destabilize brokers. Alert on filesystem utilization, partition growth, consumer lag, and the age of the oldest available record.

  • Backups: Kafka retention is not archival. Mirror important topics to durable external storage when longer-term recovery is required.

  • Security: Restrict topic-configuration changes to authorized administrators and use topic ACLs to control producer and consumer access.

  • Changes: Topic retention settings are dynamic and normally do not require a broker restart, but existing and active segments affect when data disappears.

Code Example

Set a seven-day time limit and a 10 GiB per-partition safety threshold:

# Apply topic-level retention and cleanup settings.
./bin/kafka-configs.sh \
  --bootstrap-server kafka.example.com:9092 \
  --alter \
  --entity-type topics \
  --entity-name payments \
  --add-config 'cleanup.policy=delete,retention.ms=604800000,retention.bytes=10737418240'

Verify the effective topic configuration:

# Display configuration, including dynamically overridden topic settings.
./bin/kafka-configs.sh \
  --bootstrap-server kafka.example.com:9092 \
  --describe \
  --entity-type topics \
  --entity-name payments

Monitoring and Best Practices

  • Derive retention.ms from required replay and recovery windows rather than using a generic default.

  • Treat retention.bytes as a capacity control with enough headroom to avoid premature deletion.

  • Avoid extremely short retention for audit, incident-response, or reprocessing topics.

  • Keep event-retention duration longer than the maximum expected consumer outage, including catch-up time.

  • Coordinate retention changes with offsets.retention.minutes.

  • Test deletion behavior under realistic throughput before changing a production topic.

  • Recalculate capacity whenever partition count, replication factor, traffic volume, or recovery requirements change.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 80 Apache Kafka answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Cloud, DevOps & Infrastructure subjects