Apache Kafka is a distributed, durable, high-throughput event streaming platform that acts as a replicated commit log for real-time data pipelines and messaging between systems.
Read full answerHide full answer
Applies toApache Kafka 3.3+ (KRaft production-ready; ZooKeeper removed in 4.0)
What It Is
Kafka stores streams of records in topics, which are split into partitions spread across brokers. Producers append records to partitions; consumers read them in order. Partitions are replicated for fault tolerance, and offsets track each consumer’s position, making reads replayable and decoupled from writes.
Problems It Solves
Decoupling: producers and consumers don’t need to be online or aware of each other.
Buffering and backpressure: Kafka absorbs traffic spikes so downstream systems can process at their own pace.
Durable messaging: data persists on disk with configurable retention, unlike in-memory queues.
Real-time pipelines: stream processing, log aggregation, CDC, and event-driven architectures.
Replayability: consumers can re-read historical data by resetting offsets.
Core Architecture
Broker: a Kafka server holding partitions.
Topic / Partition: logical stream split into ordered, immutable partitions.
Replication: each partition has a leader and in-sync replicas (ISR); failover promotes an ISR.
Consumer Group: consumers share partitions for parallel processing; each partition goes to one member.
KRaft: Kafka’s built-in Raft consensus for metadata management; ZooKeeper was removed in Kafka 4.0.
Guarantees and Trade-offs
Guarantee
How
Cost
Ordering
Per partition only
Must key related events to same partition
At-least-once
Default producer/consumer
Consumers must be idempotent
Exactly-once
Idempotent producer + transactions
Higher latency, transactional overhead
Durability
acks=all + replication
Higher write latency
Production Considerations
Monitoring: track consumer lag, ISR shrinkage, under-replicated partitions, and request latency via JMX/Prometheus.
Retention vs compaction: time/size retention for streams; log compaction for changelog/state topics.
Security: TLS in transit, SASL authentication, ACLs per topic; encrypt at rest via disk encryption.
Upgrades: rolling broker restarts with careful version compatibility; test client upgrades first.
Disaster recovery: MirrorMaker 2 for cross-cluster replication; back up topic configs and consumer offsets.
Common pitfalls: too few partitions (can’t scale consumers), hot partitions from poor keying, large messages, and ignoring consumer lag alerts.
When to Use
Use Kafka for high-volume, durable, replayable event streams and decoupled microservices. Prefer a simpler queue (RabbitMQ, SQS) for low-volume task queues or complex routing, and a database for transactional state.
2
How does Apache Kafka differ from RabbitMQ?
KnownRevise
Apache Kafka is a partitioned, replicated event log, while RabbitMQ is a message broker built around exchanges, queues, acknowledgements, and per-message routing. Kafka favors durable retention, replay, and high-throughput streaming; RabbitMQ favors flexible routing and conventional work-queue delivery.
Read full answerHide full answer
Applies toApache Kafka 4.x; RabbitMQ 4.x
Architecture and Data Model
Kafka stores records in topics, each divided into partitions. A partition is an append-only log with monotonically increasing offsets.
Brokers and replication: Brokers hold partition replicas. Each partition has one leader and normally one or more follower replicas.
Consumer groups: Members divide partitions between them. One group member consumes a partition at a time, and ownership changes through rebalancing.
KRaft: Kafka 4.x uses a KRaft controller quorum for metadata and partition-leader management rather than ZooKeeper.
Offsets: Consumers track their progress with committed offsets, allowing another consumer or group to replay the log independently.
RabbitMQ routes messages through exchanges into queues. Consumers receive queued messages and acknowledge them; unacknowledged messages can be redelivered. Exchanges support direct, topic, fanout, and headers-based routing.
Delivery, Ordering, and Offsets
Kafka delivery semantics depend on application and producer configuration:
At-most-once: Commit offsets before processing; a failure can lose unprocessed work.
At-least-once: Process first and commit afterward; a failure between those steps can cause duplicate processing.
Exactly-once within Kafka: For consume-transform-produce, use a transactional producer, include consumed offsets in the same transaction, and configure consumers with isolation.level=read_committed. This does not make arbitrary external side effects exactly-once.
Ordering: Kafka preserves order within a partition, not across a topic. Related records require the same key and an unchanged partition count to remain on the same partition.
RabbitMQ normally uses explicit acknowledgements, negative acknowledgements, prefetch limits, and redelivery. Publisher confirms indicate that a broker accepted a publish; they do not confirm consumer processing.
For reliable Kafka processing, disable automatic offset commits and commit only after successful handling. enable.idempotence=true with acks=all prevents retry duplicates and protects acknowledged writes from broker retries.
Feature Comparison
Area
Apache Kafka
RabbitMQ
Primary model
Replicated partitioned log
Exchanges, queues, and acknowledgements
Consumption
Consumer groups and committed offsets
Competing consumers and acknowledgements
Message lifecycle
Retained until retention or compaction removes it
Removed after acknowledgement; unacknowledged messages may be redelivered
Routing
Topic plus key or partition selection; no native exchange-style routing
Direct, topic, fanout, and headers exchanges
Ordering
Per partition
Queue insertion order per publishing channel; completion order can differ with concurrent consumers
Delivery
At-most-once, at-least-once, or transactional exactly-once within Kafka
At-most-once or at-least-once consumption
Primary scaling
Partitions and independent consumer groups
Queue distribution and consumer concurrency
Retention, Compaction, and Replay
Kafka applies retention policies independently to topics:
delete: Removes old segments according to age or size limits.
compact: Eventually retains the latest record for each key and removes older records with that key.
compact,delete: Combines compaction with normal retention.
This replayable history supports event sourcing, state rebuilding, and reprocessing. RabbitMQ queues are primarily delivery structures rather than long-term archives. Durable queues survive broker restart, while quorum queues replicate messages across RabbitMQ nodes for higher availability.
Operations, Failures, and Security
Monitor Kafka consumer lag, under-replicated and offline partitions, broker disk utilization, request latency, controller health, and JVM or garbage-collection pressure. Consumer lag is generally more useful than queue depth because Kafka deliberately retains consumed records.
Broker loss: The KRaft controller elects an in-sync replica as leader. Maintain adequate replication-factor and min.insync.replicas; without an in-sync replica, an unclean election may lose data.
Disk exhaustion: Retention can fall behind, so alerts should forecast disk capacity rather than rely only on current usage.
Rebalances: Long processing or frequent membership changes interrupt progress. Bound processing and use cooperative assignment where appropriate.
Hot partitions: A skewed key can overload one partition.
Poison records: Use bounded retries and route repeatedly failing records to a dead-letter topic.
Kafka replication provides availability, not backup. Use a tested snapshot/export or object-store backup process for logical recovery; MirrorMaker 2 provides cross-cluster replication, but replicated deletions or corruption can also propagate. Rolling upgrades must follow Kafka’s supported compatibility sequence.
Secure Kafka with TLS, SASL such as SCRAM-SHA-512, and topic ACLs. Monitor RabbitMQ queue depth, unacknowledged deliveries, memory or disk alarms, node health, and quorum-queue failover; RabbitMQ also provides TLS, authentication, and authorization controls.
Ecosystem and Choosing One
Apache Kafka provides Kafka Connect for source and sink connectors and Kafka Streams for stateful stream-processing applications. Connect exactly-once support remains connector-specific. Schema Registry is separate from Apache Kafka; Confluent Schema Registry or a compatible implementation manages Avro, JSON Schema, and Protobuf schemas.
Choose Kafka when events must be retained, replayed, partitioned across many consumers, or integrated with stream processing. Choose RabbitMQ when flexible routing, acknowledgements, priorities, dead-letter exchanges, request/reply patterns, and conventional task queues matter more than long-term log retention.
3
How do Kafka brokers, topics, and partitions relate?
KnownRevise
In Apache Kafka, a broker stores and serves Kafka data, a topic is a logical stream of records, and a partition is the ordered, append-only log within a topic. Together, they determine how data is distributed, replicated, consumed, and scaled.
Read full answerHide full answer
Applies toApache Kafka 3.9+ (KRaft mode)
Core Components
Component
Responsibility
Production implications
Broker
A Kafka server that stores partition data and serves producer and consumer requests
Multiple brokers provide capacity and failure isolation; each broker is a separate failure domain
Topic
A named category or stream of records, such as orders or payment-events
Topics are logical; their data is physically stored in partitions
Partition
An ordered sequence of records identified by monotonically increasing offsets
Partitions are the unit of parallelism, replication, retention, and consumer assignment
Leader
The broker replica currently accepting writes and serving that partition
Producers send records to the leader, which replicates them to followers
Followers
Broker replicas that maintain copies of the partition
If the leader fails, an in-sync follower can become the new leader
A topic such as orders might contain 12 partitions spread across three brokers. Each partition is replicated three times, so the topic has a replication factor of three and 36 physical replica directories across the cluster, excluding any additional recovery or archival copies.
Partitions, Ordering, and Consumer Groups
Kafka guarantees record ordering only within a partition. It does not provide a global ordering across partitions.
Partition selection: A producer sends keyed records to the same partition when the partition count and key remain stable. The producer hashes the key using the configured partitioner.
Parallel processing: Different consumer-group members process different partitions of the same topic, allowing processing to scale with partition count.
Consumer offsets: Each consumer group tracks the next position to read for every assigned partition. These offsets are stored in Kafka’s internal __consumer_offsets topic.
Rebalancing: When members join, leave, or exceed the configured max.poll.interval.ms, Kafka reassigns partitions. Frequent rebalances can cause consumer lag or duplicate processing.
Partition count: Increasing it can increase throughput but changes the partitioning of keyed records. Kafka does not support decreasing a topic’s partition count.
To preserve ordering, producers should send related records to the same partition rather than trying to coordinate ordering across Kafka.
Replication and Delivery Guarantees
A typical production configuration uses at least three replicas:
replication.factor=3: Store each partition on three brokers.
min.insync.replicas=2: Require two in-sync replicas for a successful write.
Producer acks=all: Wait for the leader and required followers to acknowledge the record.
With three replicas and min.insync.replicas=2, the topic remains writable if one broker fails. If two replicas leave the in-sync replica set, writes should fail instead of silently reducing durability.
Kafka’s at-least-once processing is achieved by committing a consumer offset only after processing succeeds. A crash between processing and committing causes the record to be replayed. Exactly-once behavior requires additional mechanisms: an idempotent producer, Kafka transactions where applicable, isolation.level=read_committed on consumers, and exactly-once-capable source connectors. Exactly-once Kafka transactions do not automatically make arbitrary external side effects exactly once.
Retention and Compaction
Kafka manages storage primarily through log segments rather than deleting individual records immediately.
Time retention: retention.ms or retention.bytes causes old segments to be deleted.
Size retention: The oldest segments are removed when the configured byte limit is exceeded.
Compaction: With cleanup.policy=compact, Kafka retains the latest record for each key rather than every historical value.
Tombstones: A record with a null value marks a key as deleted, but the tombstone itself remains until retention or compaction removes it.
Use compaction for state-like streams where only the latest value matters. Use normal retention for immutable event logs. Configure retention according to replay, recovery, and regulatory requirements rather than simply filling available disk.
Kafka Ecosystem
Kafka Connect runs source and sink connectors for moving data between Kafka and systems such as databases, object stores, and search engines. Connector concurrency scales independently from broker capacity.
Kafka Streams is a Java library for processing streams inside an application. It supports stateful operations, local state stores, and changelog topics.
Schema Registry stores and validates schemas, commonly Avro, Protobuf, or JSON Schema. Schema compatibility rules prevent producers from introducing incompatible record shapes.
KRaft replaces ZooKeeper for cluster metadata. In modern Kafka clusters, dedicated controller nodes form the metadata quorum, while brokers store data and serve clients.
Operations, Security, and Recovery
Inspect topology with kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders. This shows the leader, replicas, in-sync replicas, and partition assignments. Consumer health can be checked with kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processors.
Monitor under-replicated and offline partitions, ISR changes, producer errors, request latency, disk usage, JVM pressure, controller health, and consumer lag. Alert on sustained ISR shrinkage rather than treating every transient broker restart as a data-loss incident.
Replication is not a backup: it can replicate accidental deletions or corruption. Use cross-cluster replication, an appropriate Kafka Connect sink, or object-storage archival for disaster recovery. Protect Kafka with TLS and SASL, least-privilege ACLs, encrypted storage where required, and separate listener configurations for clients, brokers, and controllers. Perform broker upgrades as rolling operations, verify client compatibility, and confirm replication health before continuing each stage.
4
What is an Apache Kafka cluster?
KnownRevise
An Apache Kafka cluster is a distributed system of Kafka brokers that stores ordered, replicated event streams in topics. Producers publish records to partitions, while consumers read them independently or as members of consumer groups.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.x with KRaft mode
Core Architecture
A production Kafka cluster normally separates two responsibilities:
Brokers: Store partition data, serve producer and consumer requests, replicate records, and perform leader elections.
KRaft controllers: Maintain cluster metadata such as topics, partitions, replication assignments, and broker membership. A quorum of controllers prevents conflicting metadata changes.
A topic is divided into partitions, which are the unit of parallelism and storage. Each partition has one leader broker and zero or more follower brokers.
Component
Purpose
Producer
Sends records to a topic, optionally using keys for partition selection
Broker
Stores partitions and serves client requests
Partition
Append-only, ordered log segment
Consumer
Reads records from assigned partitions
Consumer group
Distributes partitions among consumers and coordinates offset commits
KRaft controller quorum
Manages and distributes cluster metadata
Clients initially contact a broker through the bootstrap.servers setting. That broker returns metadata containing the leaders for the required partitions, after which clients connect directly to the appropriate leaders.
Replication and Durability
Each partition is replicated across multiple brokers. Acknowledgement settings and min.insync.replicas determine whether a write is considered safe.
For strong durability:
Use at least three brokers for important topics.
Set an appropriate replication factor, commonly 3.
Configure min.insync.replicas to 2.
Use acks=all for critical producer writes.
Ensure the producer can handle retriable errors and configure delivery timeouts appropriately.
With acks=all, if the in-sync replica count falls below min.insync.replicas, produce requests fail because too few replicas are in sync. Using acks=1 does not provide this guarantee because the leader can acknowledge a write after appending it locally without waiting for a follower.
When a leader fails, a follower in the in-sync replica set can become the new leader. Replication protects against broker or disk failure, but not against deleting a topic or corrupting data on every replica. Production clusters therefore also need backups, appropriate ACLs, and tested recovery procedures.
Ordering and Delivery Semantics
Kafka guarantees record ordering within a partition, not across an entire topic. Records with the same key usually go to the same partition, preserving their relative order.
At-most-once: The producer does not retry failed sends, and consumers commit offsets before processing. Records may be lost but should not normally be delivered twice.
At-least-once: The producer retries and consumers commit offsets only after successful processing. Duplicate processing is possible, so consumers should be idempotent.
Exactly-once processing within Kafka: Enable idempotent production (enable.idempotence=true), use a unique transactional.id, write output records and consumed offsets in the same producer transaction, and consume with isolation.level=read_committed. acks=all is also required for idempotence. This supports atomic consume-transform-produce flows but does not make arbitrary external side effects exactly once.
Consumer groups assign at most one active consumer to each partition within a group. Adding consumers helps only while partitions remain available.
Storage and Data Lifecycle
Kafka stores records in immutable segment files. Retention policies control disk usage:
Time-based retention (retention.ms): Records become eligible for deletion after the configured age. Kafka deletes eligible rolled segments; it does not immediately remove records from an active segment.
Size-based retention keeps a bounded amount of data per partition.
Log compaction retains the latest record for each key, useful for state-like topics.
Delete retention controls how long compaction tombstones remain available to remove older records.
Partition count determines maximum consumer-group parallelism and influences broker load. Increasing it adds partitions and can change the default key-to-partition mapping, so partition growth should be planned carefully. Records already written remain in their original partitions.
Supporting Components and Operations
Kafka Connect runs source and sink connectors for moving data between Kafka and systems such as databases, search indexes, and cloud services.
Kafka Streams builds stateful or stateless stream-processing applications directly in Java or other supported runtimes.
Schema Registry manages schemas and compatibility, commonly with Avro, Protobuf, or JSON Schema.
Production monitoring should cover under-replicated partitions, offline partitions, leaders without followers, request latency, consumer lag, disk usage, controller quorum health, JVM pressure, and broker connectivity. Security should use TLS encryption and SASL authentication with topic-level ACLs rather than relying on network isolation alone.
Kafka upgrades should follow the supported rolling-upgrade procedure. Before upgrading, verify broker/controller compatibility, listener configuration, JVM requirements, and the target release’s migration requirements. In KRaft mode, format the metadata quorum only during initial cluster creation or the prescribed metadata migration—not on every restart.
5
How does KRaft replace ZooKeeper in Apache Kafka?
KnownRevise
KRaft (Kafka Raft) is Kafka’s built-in consensus protocol that stores cluster metadata in an internal replicated log instead of an external ZooKeeper ensemble, removing ZooKeeper as a dependency entirely.
Read full answerHide full answer
Applies toApache Kafka 3.3+ (KRaft production-ready for new clusters); ZooKeeper removed in Kafka 4.0.
What KRaft Is
KRaft implements the Raft consensus algorithm inside Kafka itself. Cluster metadata (topics, partitions, configurations, ACLs, broker registrations) is written to a special internal topic, __cluster_metadata, which is replicated across a set of controller nodes forming a quorum. One controller is elected active leader; the rest are hot standbys. This makes metadata management a first-class part of Kafka rather than an external service.
How It Replaces ZooKeeper
ZooKeeper previously held the source of truth for broker membership, topic metadata, controller election, and configuration. KRaft moves all of that into Kafka’s own replicated log:
Controller election happens via Raft leader election among controller quorum voters, not via ZooKeeper ephemeral nodes.
Metadata propagation uses the replicated metadata log; brokers fetch metadata updates from the active controller like they would fetch topic data.
Broker registration is recorded in metadata records rather than ZooKeeper znodes.
Single security model: one TLS/SASL configuration for the whole cluster instead of separate ZooKeeper security.
Architecture and Components
Controller quorum: a small set of nodes (typically 3 or 5) running in controller or combined mode, voting on metadata changes.
Metadata log: __cluster_metadata, an internal topic with its own partitions and replication, snapshotted periodically for fast recovery.
Broker nodes: run in broker mode and receive metadata from controllers; in small deployments a node can run in combined mode as both broker and controller.
KRaft mode flag: process.roles=broker,controller (combined) or split roles for production.
Operational Differences
Aspect
ZooKeeper mode
KRaft mode
External dependency
ZooKeeper ensemble required
None
Controller failover
Seconds to tens of seconds
Sub-second to few seconds
Metadata scalability
ZK znode limits
Log-based, much larger scale
Security config
Separate ZK auth
Unified Kafka auth
Upgrade path
In-place migration supported
Target state for Kafka 4.0
Migration and Compatibility
Existing ZooKeeper-based clusters can migrate to KRaft using the metadata migration tooling, including kafka-metadata-quorum.sh for quorum status, with the migration process documented for Kafka 3.5+. Kafka 4.0 removed ZooKeeper support entirely, so new clusters should be deployed in KRaft mode from the start. Snapshots in the metadata log make controller restarts and recovery significantly faster than ZooKeeper-based recovery.
Best Practices and Pitfalls
Use an odd number (3 or 5) of controller voters for quorum; never run a single controller in production.
Separate controller and broker roles on distinct nodes for large clusters to isolate metadata traffic.
Keep controller.quorum.voters identical across all nodes; mismatches prevent quorum formation.
Monitor kafka.controller metrics and metadata log lag; a lagging follower controller delays failover.
Do not mix ZooKeeper and KRaft metadata in the same cluster; migration is one-directional and must complete fully.
Back up by replicating the metadata log via normal Kafka replication; there is no separate ZooKeeper backup to manage, but snapshot directories under log.dirs should be included in node backups.
6
What is an offset in Apache Kafka?
KnownRevise
An offset in Kafka is a monotonically increasing integer assigned to each record within a partition, serving as its position in that partition’s append-only log. Consumers track their progress by committing offsets, allowing a consumer group to resume from its last committed position after restarts, rebalances, or failures.
Read full answerHide full answer
Applies toApache Kafka 4.x (KRaft mode)
How It Works
Each partition is an independent append-only log. Records normally begin at offset 0, and clients do not choose offsets; the broker assigns them as records are appended.
Offsets are per partition, not per topic. A record is uniquely identified by the tuple (topic, partition, offset).
The log end offset (LEO) is the next offset that will be assigned. The high watermark (HW) marks the latest record fully replicated to all current in-sync replicas.
The default isolation level, read_uncommitted, can expose records through the LEO. A read_committed consumer instead stops at the last stable offset (LSO), excluding records from transactions that have not completed.
Offsets are immutable positions. Retention deletion and log compaction can leave gaps, but surviving records are not renumbered.
KRaft replaces ZooKeeper-based controller metadata management; it does not change offset semantics. Consumer-group offsets are still stored in Kafka’s internal __consumer_offsets topic.
Consumer Offsets and Committing
The __consumer_offsets topic stores the committed offset for each consumer group and topic-partition.
A committed offset represents the next record to consume. Committing offset 42 means that consumption should resume at 42, after processing through 41.
Auto-commit is controlled by enable.auto.commit, whose default is true, and commits positions returned by consumer polls in the background. Java consumers can use commitSync() or commitAsync() for manual control.
The broker setting offsets.retention.minutes controls how long offsets are retained; its default is 10080 minutes, or seven days. If a group does not commit an offset during that period, the offset can expire.
When a committed offset is missing or out of range, the consumer initializes according to auto.offset.reset: latest by default, or earliest when configured.
Delivery Semantics and Ordering
Kafka preserves record order within a partition, and consumers observe those records in offset order. It provides no ordering guarantee across partitions.
At-least-once processing is obtained by committing only after successful processing. A crash after processing but before committing can cause the same record to be processed again.
Exactly-once processing within Kafka requires a transactional producer with a stable transactional.id, transactional publication of output records and consumer offsets using sendOffsetsToTransaction(), and consumers configured with isolation.level=read_committed. Idempotence alone is not sufficient, and transactions do not atomically cover external database or HTTP side effects.
Operational Considerations
Consumer lag is a derived metric rather than a value stored by the broker. It is commonly calculated from the group’s committed or current position and the partition’s log-end position. During active processing, committed lag can overstate the consumer’s in-process lag; for read_committed consumers, the LEO can also overstate useful application-record lag because it includes control records and records from incomplete transactions.
Inspect committed offsets and lag with a runnable command such as bin/kafka-consumer-groups.sh --bootstrap-server broker.example.com:9092 --describe --group my-group.
Reset offsets only while the group has no active members. For example: bin/kafka-consumer-groups.sh --bootstrap-server broker.example.com:9092 --group my-group --topic my-topic --reset-offsets --to-earliest --execute. The tool supports --to-earliest, --to-latest, --to-datetime, and --shift-by.
A committed offset below the partition’s log start offset is out of range after retention deletion. The consumer handles it according to auto.offset.reset; if the policy is none, consumption cannot proceed until an operator supplies a valid position.
On compacted topics, offsets are not contiguous and therefore cannot be interpreted as record counts.
Best Practices
Use manual commits after successful processing to define replay boundaries, while accepting that duplicates remain possible between processing and commit.
Make consumers idempotent, or use a stable event identifier for deduplication, when downstream side effects may be repeated.
Alert on per-partition lag and the age of the oldest unprocessed record rather than relying only on aggregate lag.
Keep offsets.retention.minutes above the longest expected period of consumer inactivity when uninterrupted group resumption is required.
Use Kafka transactions and read_committed only when end-to-end exactly-once behavior is required, and design separately for external side effects that Kafka transactions cannot cover.
7
How does Apache Kafka store messages on disk?
KnownRevise
Kafka stores messages as append-only partition logs organized into immutable log segments. Each topic is split into partitions, and every partition replica maintains its own on-disk log containing message batches, offset indexes, and timestamp indexes.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode
Disk Layout and Record Format
A broker stores partition data beneath the directories configured by log.dirs. Each partition replica normally has a directory named using the topic name, partition number, and a unique replica identifier.
A partition contains files such as:
Log segment (.log): Stores compressed message batches in append-only order.
Offset index (.index): Maps logical offsets to physical positions in log segments, allowing efficient lookup without scanning the entire segment.
Timestamp index (.timeindex): Provides a sparse timestamp-to-offset mapping for timestamp-based offset lookups.
Leader epoch checkpoint (leader-epoch-checkpoint): Records changes in partition leadership, helping followers validate leader history during replication.
Messages are written as record batches. A batch contains multiple records and metadata such as the base offset, timestamps, producer ID and epoch, base sequence, leader epoch, compression settings, and a CRC32C checksum. Compression can be applied to the records within a batch, reducing disk usage and network transfer at the cost of additional CPU usage.
Kafka uses the operating system’s page cache to buffer writes. It does not normally perform an fsync for every producer request; replication protects against the loss of an individual replica, while the operating system controls when dirty pages reach persistent storage.
Partitioning and Replication
Messages within a partition retain their append order, but Kafka provides no ordering guarantee across different partitions. A producer chooses the destination partition, usually by key hashing; the same key therefore maps consistently to one partition unless the partition count changes or a custom partitioner behaves differently.
Every partition has one leader and zero or more follower replicas. Only the leader accepts writes, and followers replicate those batches. With a replication factor of three, a complete copy of each partition is generally maintained on three brokers.
KRaft stores Kafka’s cluster metadata, including topic and partition leadership information, in a separate internal metadata log. It does not replace the per-partition message logs.
Retention and Log Compaction
Kafka controls disk usage with two primary topic-level mechanisms:
Delete retention: cleanup.policy=delete removes whole log segments after the configured age or size limits expire. Active segments are normally rolled before they can be removed, so actual retention can exceed the configured value.
Log compaction: cleanup.policy=compact removes older records when a newer record with the same key exists in the same partition. The latest value for a key is not necessarily removed immediately, and compaction does not guarantee that exactly one physical record remains for every key.
Important retention configurations include:
retention.ms: Maximum age of records.
retention.bytes: Maximum retained data per partition.
segment.bytes: Maximum size of a log segment before rolling.
segment.ms: Maximum age of a log segment before rolling.
cleanup.policy: Controls deletion, compaction, or both.
A record with a null value is a tombstone. During compaction, it removes the previous non-null value for that key, but the tombstone itself remains until delete-retention processing removes it.
Consumer group offsets are stored separately in the internal __consumer_offsets topic, which is compacted and replicated. Business-message storage and consumer offset storage should not be treated as the same mechanism.
Durability and Failure Modes
For production durability, use a replication factor of at least three, configure producers with acks=all, and set min.insync.replicas appropriately. For example, with three replicas, min.insync.replicas=2 rejects writes when fewer than two replicas are available.
Important failure scenarios include:
Broker loss: A follower can be recreated from the leader. If the leader fails, another in-sync replica can become leader.
Unclean leader election: Keep unclean.leader.election.enable=false to prevent an out-of-sync replica becoming leader and silently losing records.
Disk exhaustion: Retention failures, oversized segments, compaction backlog, or insufficient disk can stop writes. Capacity should account for replication: three replicas require approximately three times the retained message data.
Corrupted segment: Kafka validates record checksums, but operators still need tested procedures for restoring or rebuilding affected replicas.
Long garbage-collection or I/O pauses: These can increase producer latency, request timeouts, and consumer lag.
Operational Practices
Inspect partition health with commands such as bin/kafka-topics.sh --bootstrap-server 10.0.0.10:9092 --describe --topic orders --under-replicated-partitions. Review topic retention with bin/kafka-configs.sh --bootstrap-server 10.0.0.10:9092 --describe --entity-type topics --entity-name orders.
Production monitoring should include:
Disk usage, free space, and I/O latency per broker.
Under-replicated and offline partition counts.
Broker log flush errors and failed requests.
Producer retry and delivery-failure rates.
Consumer lag and offsets for critical consumer groups.
Retention deletion and compaction backlogs.
Plan capacity using peak ingestion, replication factor, retention, compression, and segment overhead. Before upgrades or broker replacements, verify that replicas are healthy and that the replacement broker has sufficient disk and network capacity.
8
What are log segments in Apache Kafka?
KnownRevise
A log segment is a file-backed segment of an Apache Kafka partition’s append-only log. Each partition consists of one or more segments, allowing Kafka to roll active data into separate files and reclaim space without rewriting the whole partition.
Read full answerHide full answer
Applies toApache Kafka 3.6–4.x
How Segments Work
A topic is divided into partitions, and each partition has an independent ordered log.
The active segment is the file to which the broker currently appends record batches.
Kafka rolls the active segment when a configured size or time threshold is reached and starts a new segment.
A segment filename begins with the segment’s base offset, the offset of its first record. Kafka normally zero-pads this value to 20 digits, for example 00000000000000000123.log.
Companion .index and .timeindex files map offsets and timestamps to physical positions, allowing Kafka to locate records without scanning an entire segment.
The index is generally sparse, so it does not contain an entry for every record.
Segments are local files on each broker. Leaders append records, while follower replicas maintain corresponding local log data.
Segments are an internal storage detail. Producers write to partitions, and consumers fetch records by offset; neither normally interacts with segment files directly. Crossing a segment boundary does not create a transaction boundary or change delivery semantics.
Retention and Compaction
Kafka normally applies retention at segment granularity rather than deleting individual records.
cleanup.policy=delete: Removes old segments when the configured time or size threshold is reached. If both are configured, either threshold can trigger deletion.
log.retention.ms: Maximum time that log data is retained when time-based retention applies.
log.retention.bytes: Maximum retained size per partition when size-based retention applies.
log.segment.bytes: Segment-size threshold used when rolling the active segment.
log.roll.ms: Time-based threshold for rolling the active segment.
log.roll.jitter.ms: Randomizes time-based segment rolls, reducing synchronized roll activity across partitions.
With cleanup.policy=compact, the log cleaner removes obsolete records for keys that it processes. The compacted state exposes the latest value for each key, although Kafka does not guarantee that every physical segment contains at most one record for that key.
A null-valued record is a tombstone. It causes older values for that key to be removed during compaction and marks the key as deleted for compacted-state consumers. The tombstone itself may be physically removed only after the configured delete.retention.ms interval.
Compaction operates within a partition. Records for a key must therefore use the same partition for compaction to provide the intended result.
Delivery Guarantees
Kafka’s ordering guarantee applies within a partition, including across segment boundaries. There is no ordering relationship between different partitions.
Mechanism
Guarantee and requirements
At-least-once consumption
Process the record before committing its offset. Disable automatic commits or otherwise commit only after successful processing. A crash after processing but before committing can cause reprocessing.
Idempotent production
enable.idempotence=true prevents retry duplicates within the producer’s idempotence session. It does not provide end-to-end exactly-once processing.
Transactional exactly-once
For Kafka consume-transform-produce flows, use a transactional producer with a unique transactional.id, send records and consumer offsets in the same transaction, and call commitTransaction(). Consumers must use isolation.level=read_committed to avoid observing uncommitted or aborted transactional records.
A segment boundary does not affect any of these guarantees.
Operational Considerations
Disk pressure: Partition count, replication, retention, and delayed segment cleanup can exhaust broker disks. log.segment.bytes controls segment rolling, not total partition storage.
Consumer lag: Retention may remove records before a lagging consumer reads them. If its committed offset is below the log-start offset, offset-reset behavior can move it forward and silently skip expired data.
Replication: Kafka replicates records and batches to followers over its network protocol; it does not replicate raw directory files. Monitor under-replicated and offline partitions.
KRaft: Controllers maintain cluster metadata, while partition data remains on brokers. Controller quorum health and broker disk capacity are therefore separate operational concerns.
Security: Segment files contain record data and may expose sensitive information. Restrict filesystem access, use disk encryption where required, and protect broker communication with TLS and mechanisms such as SASL.
Production Best Practices and Failure Modes
Size partitions using expected throughput, retention period, consumer parallelism, and replication factor.
Maintain sufficient disk headroom for segment rolls, compaction, replicas, and temporary recovery activity.
Alert on disk utilization, under-replicated partitions, offline partitions, request latency, broker errors, and consumer lag.
Treat a sudden lag reduction as potentially suspicious; it may indicate expired data and an offset reset rather than healthy catch-up.
Do not manually delete, rename, or edit segment files while Kafka is running. Let Kafka perform retention and compaction, or stop the affected broker and follow Kafka’s documented recovery procedures.
9
How does replication work in Apache Kafka?
KnownRevise
Replication in Apache Kafka is the mechanism by which each partition of a topic is copied across multiple brokers so that data survives broker failures and reads/writes can continue without data loss.
Read full answerHide full answer
Applies toApache Kafka 3.7+ (KRaft mode)
How It Works
Each topic is split into partitions, and every partition is replicated across a configurable number of brokers. One replica is designated the leader; by default, all produce requests and consumer fetches for that partition go to the leader. The other replicas are followers that pull data from the leader and stay in sync. If the leader fails, one of the in-sync followers is elected as the new leader.
Key Components
Replication factor: number of copies of each partition (commonly 3 in production).
Leader: the replica that serves client traffic for a partition by default.
Followers: replicas that fetch from the leader; with rack-aware follower fetching enabled, they can also serve some consumer reads.
ISR (In-Sync Replicas): the set of replicas (leader + followers) that are caught up within replica.lag.time.max.ms.
min.insync.replicas: minimum number of replicas that must acknowledge a write when acks=all.
Durability and Acknowledgements
Producers choose durability via acks:
acks value
Behaviour
Durability
0
No acknowledgement; fire and forget
Lowest; data can be lost
1
Leader acknowledges after writing to its log
Medium; leader failure can lose unreplicated data
all (-1)
Leader waits for all ISR replicas to acknowledge
Highest; survives leader failure
With acks=all and min.insync.replicas=2 on a replication factor of 3, a write succeeds only if at least 2 in-sync replicas, including the leader, acknowledge it. If the ISR shrinks below min.insync.replicas, writes fail with NOT_ENOUGH_REPLICAS rather than silently weakening durability.
Failure Handling
If a follower falls behind beyond replica.lag.time.max.ms, it is removed from the ISR; acks=all then waits for the smaller ISR, subject to min.insync.replicas.
If the leader fails, the KRaft controller elects a new leader from the ISR. Replicas outside the ISR are not eligible, which prevents data loss but can make a partition temporarily unavailable if no in-sync follower remains.
Unclean leader election (unclean.leader.election.enable=true) allows an out-of-sync replica to become leader, trading availability for potential data loss; keep it disabled in production.
Ordering Guarantees
Ordering is guaranteed only within a single partition. Replication preserves order on the leader’s log, and followers apply records in the same order. Cross-partition ordering is not provided.
Operational Considerations
Monitoring: track UnderReplicatedPartitions, OfflinePartitionsCount, IsrShrinksPerSec/IsrExpandsPerSec, and ActiveControllerCount (should be 1).
Capacity: replication multiplies storage and network traffic by the replication factor; size brokers accordingly.
Rebalancing: adding brokers requires partition reassignment; use kafka-reassign-partitions.sh and throttle with --throttle to avoid saturating the cluster.
Backups: replication is not a backup; use MirrorMaker 2 for disaster recovery and tiered storage for long-term retention.
Best Practices
Use replication factor 3 with min.insync.replicas=2 for production topics.
Set acks=all for producers that require durability.
Place replicas across failure domains (racks/availability zones) using broker.rack and rack-aware assignment.
10
What are leader and follower replicas in Kafka?
KnownRevise
In Kafka, each partition is replicated across multiple brokers for fault tolerance. Among the replicas of a partition, exactly one acts as the leader and the rest are followers. The leader is the single replica that serves all produce and fetch requests for that partition, while followers continuously pull data from the leader to stay in sync.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x (KRaft mode)
How Replication Works
When a topic is created with replication factor N, each partition gets N replicas placed on different brokers. One replica is elected leader; the others become followers. All client traffic for that partition goes to the leader, which keeps an ordered log. Followers replicate by issuing fetch requests to the leader, appending the same records in the same order.
Leader replica: handles every produce request and every consumer fetch for its partition; advances the high-watermark as followers catch up.
Follower replica: does not serve client requests by default; pulls records from the leader and writes them to its local log; participates in leader election when the leader fails.
In-Sync Replicas (ISR)
Followers that are caught up within replica.lag.time.max.ms (default 30s) are part of the ISR set. Only ISR members are eligible to become leader, which is what makes failover safe.
min.insync.replicas: minimum ISR size required for a produce with acks=all to succeed.
If ISR shrinks below this threshold, producers with acks=all receive NOT_ENOUGH_REPLICAS errors.
Leader Election and Failover
When a leader broker fails or is shut down, the controller (the KRaft quorum leader in 3.x/4.x) elects a new leader from the ISR. The new leader’s log becomes authoritative; any records not replicated to it are lost.
unclean.leader.election.enable=false (default): only ISR members can be elected; safer, but may cause unavailability if ISR is empty.
unclean.leader.election.enable=true: allows an out-of-sync replica to become leader; risks data loss but keeps the partition available.
Follower Reads
Since Kafka 2.4, consumers can fetch from the closest replica, including followers, when client.rack is set and replica.selector.class is configured on brokers. This reduces cross-AZ traffic but does not change which replica is authoritative for writes.
Key Configuration
Setting
Scope
Purpose
Replication factor
Set at topic creation
Number of replicas per partition
min.insync.replicas
Topic or broker
Minimum ISR for acks=all writes
replica.lag.time.max.ms
Broker
How long a follower can lag before leaving ISR
unclean.leader.election.enable
Broker
Allow out-of-sync leader election
default.replication.factor
Broker
Default replication factor for auto-created topics
Production Considerations
Spread replicas across racks or availability zones using broker.rack so a single failure domain does not take out all replicas of a partition.
Monitor UnderReplicatedPartitions, OfflinePartitionsCount, and IsrShrinksPerSec/IsrExpandsPerSec; sustained ISR shrinkage indicates a slow or failing follower.
Reassign partitions with kafka-reassign-partitions.sh after adding brokers; otherwise new brokers hold no replicas and old brokers stay overloaded.
Prefer acks=all with min.insync.replicas=2 on a 3-replica topic for durability; this tolerates one broker loss without data loss.
During rolling upgrades, followers catch up after restart; watch replication lag before moving to the next broker.
11
What are in-sync replicas (ISR) in Apache Kafka?
KnownRevise
In-sync replicas (ISR) are the broker replicas of a Kafka partition that are active and sufficiently caught up with the partition leader to be trusted as synchronous copies. The leader normally belongs to the ISR, and ISR members are the preferred candidates for clean leader election.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x
ISR Membership
Each Kafka partition has one leader and one or more follower replicas:
The leader receives records from producers.
Followers fetch records from the leader and maintain local copies.
The controller tracks which replicas are sufficiently synchronized.
A follower that becomes unavailable or falls behind may be removed from the ISR.
Once the follower catches up, the controller can add it back to the ISR.
A replica can be operational but outside the ISR if it has fallen too far behind. The broker setting replica.lag.time.max.ms determines how long a replica may lag before being considered out of sync.
Replication and Acknowledgements
With acks=all, the leader waits for all replicas currently in the ISR to acknowledge the record before completing the producer request. It does not necessarily wait for every configured replica if some are outside the ISR.
The topic setting min.insync.replicas establishes the minimum acceptable ISR size:
Writes wait for all current ISR members, which may be only the leader
Greatest availability, but weakest protection if the leader’s only current copy is lost
If the ISR falls below min.insync.replicas, producers using acks=all receive errors such as NOT_ENOUGH_REPLICAS. This prevents the cluster from silently acknowledging records that exist on too few replicas.
Guarantees and Consistency
ISR supports Kafka’s replication guarantees, but it does not by itself provide exactly-once processing:
Ordering: Kafka preserves record order within a partition, not across an entire topic.
At-least-once delivery: Consumers can commit offsets before or after processing and may process a record again after a crash.
Exactly-once processing: Requires idempotent or transactional producers, Kafka transactions, and consumers using isolation.level=read_committed.
Durability: acks=all combined with an appropriate min.insync.replicas value protects against losing acknowledged records during a broker failure.
For transactional workloads, the internal transaction-state topic should also use suitable replication and ISR settings.
Failure Handling
When a leader fails, a clean election normally selects a leader from the ISR. With the broker setting unclean.leader.election.enable=false, Kafka avoids promoting a stale replica that may have lost acknowledged records.
If no ISR member is available, the choices are:
Prioritize consistency: Keep the partition unavailable until a synchronized replica returns.
Prioritize availability: Enable unclean election, allowing a potentially stale replica to become leader.
The second option can cause data loss and should be used only when the availability tradeoff is explicit.
Production Operations
Inspect the leader, configured replicas, and current ISR with:
UnderReplicatedPartitions: Partitions with a configured replica missing from the ISR.
UnderMinIsrPartitionCount: Partitions whose ISR is smaller than min.insync.replicas.
OfflinePartitionsCount: Partitions without an available leader.
ActiveControllerCount: Should normally be 1 in a healthy cluster.
During rolling upgrades, brokers restart one at a time, which can temporarily shrink the ISR. Stagger upgrades and avoid an overly aggressive min.insync.replicas, or producers may be blocked. Place replicas across separate brokers and failure domains, and do not treat ISR as a backup: ISR protects against failures within the Kafka cluster, but it does not replace off-cluster replication or independent backups. TLS, SASL, and ACLs protect access to Kafka but do not affect ISR membership.
12
How does min.insync.replicas control Kafka write durability?
KnownRevise
min.insync.replicas is the minimum number of in-sync replicas required before Kafka accepts writes that request acknowledgment from all in-sync replicas. It trades write availability for durability by preventing a producer from succeeding when too few replicas are reliably caught up.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x
How It Works
The in-sync replica set (ISR) contains the leader and replicas that are sufficiently caught up with the partition’s log.
With acks=all (represented as -1 in the protocol), Kafka checks that the ISR contains at least min.insync.replicas replicas.
If the ISR is smaller, the produce request fails with NotEnoughReplicasException.
The leader counts as one ISR replica, so a value of 2 requires the leader plus at least one caught-up follower.
The setting does not change the partition’s replication factor; it defines the minimum acceptable ISR size for protected writes.
The broker-level default is 1. A topic can override it with a stronger value such as 2 or 3.
min.insync.replicas is an availability threshold, not the number of acknowledgments the producer waits for. With acks=all, the producer waits for all replicas currently in the ISR to acknowledge the record.
Failure Behavior
Consider a topic with replication factor 3 and min.insync.replicas=2:
Condition
Result for acks=all
All three replicas are in the ISR
Writes succeed after all current ISR replicas acknowledge
One broker fails, leaving two in the ISR
Writes continue, assuming the remaining replicas are healthy
A second replica falls behind or fails
ISR falls below 2; writes are rejected
Only the leader remains in the ISR
Writes are rejected, protecting durability
If the leader fails, Kafka can elect another in-sync replica as leader. However, this setting does not disable unclean leader election. For critical data, leave unclean.leader.election.enable=false so an out-of-sync replica cannot become leader and lose committed records.
acks=1 and acks=0 do not provide the same all-replica durability guarantee. Use acks=all when min.insync.replicas is intended to enforce the configured durability floor.
Production Trade-offs
Configuration
Availability
Durability and Operational Risk
RF 3, min ISR 2
Tolerates one broker failure
Common production balance; prevents writes with only one replica
RF 3, min ISR 1
More tolerant of degraded replicas
Allows acknowledged writes with no in-sync follower copy
RF 2, min ISR 2
Very low write availability during replica loss
Requires both replicas to remain caught up, but still has little redundancy
RF 3, min ISR 3
Writes stop after any replica loss
Strongest durability threshold; unsuitable for many highly available systems
A higher value reduces the chance of acknowledging data that exists on too few replicas, but it can increase produce errors during broker maintenance, network problems, or replica lag. Place replicas across failure domains and investigate lagging replicas rather than immediately lowering the threshold.
Configuration
Set the topic-level min.insync.replicas property to the desired threshold and use a replication factor that provides adequate redundancy. A replication factor of 3 with min.insync.replicas=2 is a common starting point because it tolerates one broker or replica failure while refusing writes when only the leader remains.
The topic-level property overrides the broker-level default. Review the effective value during capacity planning and after configuration changes because raising it can make an existing topic unavailable during replica lag or broker maintenance.
Diagnose before relaxing policy: Check broker health, disk throughput, network latency, partition imbalance, and excessive consumer or producer load.
Use full acknowledgment: Configure producers with acks=all and retain idempotent production for duplicate protection.
Do not confuse it with exactly-once processing: Kafka’s exactly-once workflows use transactional production and isolation.level=read_committed; Kafka Streams provides this through processing.guarantee=exactly_once. Idempotent producers reduce retry duplicates but do not by themselves make an arbitrary producer-consumer pipeline end-to-end exactly-once.
Secure the cluster separately: TLS, SASL, and ACLs protect confidentiality and authorization but do not affect ISR membership or durability.
13
What happens when a Kafka broker fails?
KnownRevise
When a Kafka broker fails, its partitions lose their leader, and Kafka elects a new leader from the in-sync replicas so producers and consumers can continue with minimal interruption. Partitions with no in-sync replica become unavailable unless unclean leader election is explicitly enabled.
Read full answerHide full answer
Applies toApache Kafka 3.7+ (KRaft mode; ZooKeeper mode is legacy)
Leader Election and Partition Availability
Each partition has one leader and zero or more followers. Followers that are caught up form the in-sync replica set (ISR).
When the leader broker fails, the controller (in KRaft, the active controller from the metadata quorum) selects a new leader from the ISR.
Producers and consumers refresh metadata, discover the new leader, and resume. With acks=all and a healthy ISR, committed records are not lost.
If the failed broker was the only in-sync replica for a partition, that partition becomes offline until a replica catches up or unclean leader election is enabled.
unclean.leader.election.enable defaults to false in modern Kafka. Enabling it can restore availability by promoting an out-of-sync replica, but it can cause data loss and consumer offset resets.
Client and Consumer Group Impact
Producers retry failed sends according to retries and delivery.timeout.ms; idempotent producers prevent duplicate writes during retries.
Consumer groups whose coordinator was on the failed broker rediscover the coordinator on another broker and rebalance. Temporary commit failures can lead to duplicate processing, so consumers should remain idempotent.
Ordering guarantees remain per partition; a new leader continues from the committed log end, protected by leader epochs.
Recovery and Data Consistency
When the failed broker restarts, it rejoins as a follower and truncates any divergent tail using the leader epoch, then catches up from the current leader.
Replication lag and under-replicated partitions should return to normal before the cluster is considered healthy.
If the broker was also a KRaft controller, the metadata quorum elects another active controller; metadata availability depends on quorum health, not one node.
Durability and availability trade off through replication.factor, min.insync.replicas, acks, and rack awareness. A common production setting is replication factor 3 with min.insync.replicas=2 and acks=all.
Avoid treating a restarted broker as healthy until ISR membership and lag metrics recover; premature traffic shifts can increase replication pressure.
Best Practices
Keep unclean.leader.election.enable=false unless availability is explicitly prioritized over data integrity.
Use rack-aware replica placement so one failure does not remove all replicas for a partition.
Alert on sustained under-replication and offline partitions, not just broker process liveness.
Test failover in staging with controlled broker termination and verify client retry, rebalance, and recovery behavior.
14
What makes unclean leader election risky in Apache Kafka?
KnownRevise
Unclean leader election is risky because it can make a stale replica the partition leader when no in-sync replica is available, trading strict data consistency and potentially losing acknowledged records for faster recovery.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.x in KRaft mode
Failure Mechanism
Each Kafka partition has one leader and zero or more follower replicas.
The in-sync replica set (ISR) contains replicas that have caught up with the leader’s log. A normal, or clean, leader election selects only an ISR replica.
If the leader fails and no ISR replica is available, Kafka normally leaves the partition without a leader and rejects writes until a synchronized replica recovers.
With unclean.leader.election.enable=true, Kafka may instead elect an out-of-ISR replica whose log might be behind the former leader.
This affects partition leadership, not KRaft controller elections. KRaft controllers require a voting quorum and cannot safely elect a minority controller.
When divergent replicas later reconcile, Kafka can truncate log portions that conflict with the elected leader, permanently discarding records.
Data and Delivery Impact
Behavior
Clean election
Unclean election
Leader candidate
ISR replica only
Out-of-ISR replica may be selected
Availability
Partition may remain unavailable
Partition can recover if a stale replica is available
Divergent logs
Avoids selecting a known lagging replica
Can select and truncate divergent logs
Acknowledged records
May still be lost with weaker acknowledgements such as acks=1
Can be lost even when producers use acks=all
Producer acknowledgements: acks=all waits for all replicas in the current ISR, but it cannot guarantee durability if those replicas subsequently fail and only a stale replica remains eligible.
min.insync.replicas: With acks=all, this setting makes writes fail when too few replicas are synchronized. It does not prohibit an unclean election after a leader failure.
Transactions: Kafka’s exactly-once transaction flow can atomically commit a consume-transform-produce operation only while the required records remain in the log. It cannot restore transactional records missing from the newly elected leader.
Ordering: Ordering remains defined per partition for records that survive. Unclean election can nevertheless create logical gaps and discard records, so it does not preserve a complete ordered stream.
Consumer groups: Consumer offsets are not automatically repaired to reflect the truncated log. Consumers can encounter offset-out-of-range behavior or continue from an offset that no longer corresponds to the same data.
Operational Trade-offs
Unclean election prioritizes partition availability over consistency. It may be acceptable in an explicitly lossy recovery scenario, but the organization must accept that acknowledged data can disappear.
A typical replicated deployment uses a replication factor of at least three, min.insync.replicas=2, producer acks=all, and idempotent producers. These settings reduce exposure but do not make unclean election safe.
Monitor UnderReplicatedPartitions, OfflinePartitions, leader-election activity, broker errors, and controller or broker logs for unclean-election events.
During rolling upgrades, upgrade one broker at a time and verify that ISR membership and replica health recover before continuing.
Replication is not a backup. Mirror critical topics to an independent cluster and regularly test restoration procedures.
Production Practices
Keep unclean.leader.election.enable=false on every broker. This is the default read-only broker setting, so changing it requires a controlled configuration rollout and broker restart.
Investigate the original broker crash, network partition, disk failure, or failed upgrade rather than repeatedly forcing leader changes.
If an unclean election occurs, pause or rate-limit affected producers, preserve broker and controller logs, compare partition offsets, and identify potentially missing records.
Restore adequate ISR capacity and confirm synchronization before declaring the incident resolved. Avoid manually promoting a stale replica unless the same acknowledged-data-loss risk is explicitly accepted.
Protect broker, controller, and client connections with TLS and SASL authentication. ACLs limit unauthorized access but do not change the consistency consequences of an unclean election.
15
What does an Apache Kafka producer do?
KnownRevise
Kafka producer is a client application that publishes records to Kafka topics. It selects partitions, batches records for efficient transmission, handles broker acknowledgments and retries, and provides delivery behavior according to its reliability configuration.
Read full answerHide full answer
Applies toApache Kafka 4.x
How It Works
ProducerRecord: A producer sends a key, value, and optional headers to a topic. Kafka stores the record in the selected partition.
Partition selection: The producer obtains topic metadata, chooses a partition, and sends the request to that partition’s leader. The default partitioner generally maps a key to a consistent partition, while records without keys can be distributed across partitions to reduce hotspots.
Batching: The RecordAccumulator groups records into batches. The producer’s sender thread transmits these batches asynchronously, reducing network overhead and increasing throughput.
Replication: The leader broker appends the record and replicates it to followers in the partition’s in-sync replica set, or ISR.
KRaft: Kafka 4.x supports KRaft rather than ZooKeeper. KRaft controllers maintain the cluster metadata log, while partition leaders handle record writes.
Kafka preserves record order within a partition, not across an entire topic. Increasing a topic’s partition count can change key-to-partition mappings, so applications requiring long-term ordering need an explicit partitioning strategy or a stable event identifier.
Delivery Guarantees
Configuration
Acknowledgment behavior
Main trade-off
acks=0
The producer does not wait for a broker acknowledgment.
Lowest latency, but send failures may not be reported to the application.
acks=1
The leader acknowledges after appending the record to its local log.
A leader failure before replication can lose the record.
acks=all
The leader waits for all current in-sync replicas, including itself, to acknowledge the record.
Stronger durability, but latency depends on ISR health.
At-least-once delivery: Use acks=all and allow retries. If a request times out, the producer cannot know whether the broker committed the record; retrying it can therefore create a duplicate.
Idempotent delivery: The Kafka 4.x client enables idempotence by default. It prevents duplicate records caused by retries within the same producer session and partition sequence. Setting enable.idempotence=false explicitly disables it; incompatible acks, retries, or max.in.flight.requests.per.connection values are rejected with a configuration error rather than silently overriding idempotence.
Exactly-once processing: Idempotence alone does not make an entire application workflow exactly once. Kafka Streams exactly-once processing uses Kafka transactions to atomically publish output records and commit the corresponding consumed offsets. For an equivalent direct-client workflow, write records and offsets through the same transaction and use isolation.level=read_committed to read only committed records.
Ordering: Idempotent producers preserve partition order across retries. Without idempotence, multiple in-flight requests combined with retries can reorder records; setting max.in.flight.requests.per.connection=1 avoids that scenario at the cost of throughput.
Performance and Operations
Tune batch.size, linger.ms, and compression.type for throughput, while monitoring end-to-end latency rather than maximizing batch size indiscriminately.
Reuse a long-lived KafkaProducer; it is thread-safe and maintains metadata, batching, and connection state.
delivery.timeout.ms bounds the total time allowed for a send, including retries and waiting for the broker response. request.timeout.ms controls the timeout for an individual broker request.
Monitor producer record-error and retry rates, batch size, queue time, request latency, metadata age, and throughput.
Monitor brokers for under-replicated partitions, shrinking ISR, disk utilization, request latency, and garbage-collection pressure.
Kafka Connect sink connectors use producers internally, while Kafka Streams applications use producers directly, including transactional producers for exactly-once processing.
Confluent Schema Registry is a separate service supporting Avro, Protobuf, and JSON Schema. Schema or payload validation occurs only when compatible serializers or connectors are configured to perform it; it is not automatic behavior of the Apache Kafka producer.
Security and Failure Modes
Use TLS for encryption in transit and SASL authentication such as SCRAM-SHA-512 for client identity.
Grant only required topic-write and transactional-ID permissions through Kafka ACLs. Keep credentials outside source control and inject them from a secret manager.
Common incidents include metadata timeouts, unavailable leaders, disk exhaustion, prolonged ISR shrinkage, oversized records, and incompatible schemas.
An acks=all producer can still fail when a partition lacks enough healthy replicas. Configure min.insync.replicas, maintain sufficient broker redundancy, and plan capacity accordingly.
Production Best Practices
Pre-create topics instead of depending on broker-side automatic topic creation.
Keep acks=all, idempotence, and suitable retry limits enabled for important events.
Give every concurrently running transactional producer a unique transactional.id; reuse that ID across restarts of the same instance for fencing and recovery, but never share it between active instances.
Validate schemas before publication and evolve them compatibly.
Treat successful send completion—not merely a successful send() call—as evidence that the broker accepted the record.
16
How does a Kafka producer choose a message's partition?
KnownRevise
A Kafka producer selects a message’s partition on the client. With the Kafka 4.x Java client’s built-in partitioner, an explicitly specified partition takes priority; otherwise, a non-null key is mapped using murmur2-compatible hashing, while a null key uses sticky partition assignment.
If it does not exist, the producer reports an invalid-partition error rather than selecting another partition
Non-null key
Hashes the key and applies modulo to the topic’s current partition count
The same key routes to the same partition while the partition count remains unchanged and the built-in partitioner is used
Null key
Uses sticky assignment for the current producer batch
There is no stable key-to-partition relationship across batches
Custom partitioner
Uses the implementation configured by partitioner.class when no explicit partition is supplied
It can implement business-specific routing, but must be tested against partition growth and failure scenarios
With the built-in partitioner, adding partitions changes the modulo calculation and can remap some keyed records. Historical and new records for the same key may therefore reside in different partitions, breaking assumptions about colocated ordering or compaction.
How Selection Works
The producer retrieves metadata for the topic, including its partitions and leaders.
It selects the partition locally according to the record and the configured partitioner.
It batches records and sends each batch to the selected partition’s leader.
After a leader change or a retryable failure, the producer refreshes metadata and retries the request as permitted by its configuration.
The returned Future or completion callback reports the assigned partition and offset on success, or the delivery error on failure.
A successful call to send() only means that the record was accepted into the producer’s local batching process; it does not prove that the broker durably appended it. Applications should handle future or callback errors and monitor producer error and retry metrics.
Delivery Guarantees and Failure Modes
Ordering: Kafka preserves log order within a partition, not across a topic. The built-in partitioner maps a given key consistently, but disabling idempotence while allowing multiple requests in flight can permit retry reordering. Kafka 4.x enables idempotence by default.
At-least-once delivery: acks=all, retries, replication, and an appropriate min.insync.replicas configuration provide at-least-once producer behavior, subject to the durability of the replicated cluster. Retries can duplicate records when idempotence is disabled.
Idempotent retries: Kafka 4.x defaults enable.idempotence to true. It requires acks=all, a positive retry limit, and max.in.flight.requests.per.connection no greater than 5; it prevents duplicates caused by producer retries within a producer session.
Exactly-once processing: Kafka transactions are required for an atomic consume-transform-produce workflow. Use a stable, instance-specific transactional.id and configure consumers with isolation.level=read_committed. External database or HTTP side effects still require idempotency, an outbox, or an equivalent design.
Hot partitions: A small number of keys can send disproportionate traffic to one partition. Partition count alone does not correct a hot-key workload.
Explicit partitioning: Pinning records to a partition creates coupling to the current partition layout and can concentrate load. Missing partitions cause send failures rather than automatic repartitioning.
Production Practices
Inspect the partition count and leaders with kafka-topics.sh --bootstrap-server kafka-1.example.com:9092 --describe --topic orders.
Monitor per-partition traffic, producer errors and retries, request latency, under-replicated partitions, and consumer lag.
Treat partition growth as a data-routing operation: it can remap keys and separate records that are expected to be compacted together.
Use custom partitioners only when business-specific routing is necessary. Test them with null keys, hot keys, partition growth, leader changes, and broker failures.
Protect producer access with TLS or SASL and appropriate ACLs. Partition selection is a routing mechanism, not a security boundary.
17
What are Kafka message keys and why do they matter?
KnownRevise
In Apache Kafka, a message key is an optional byte array attached to each record that the producer uses, by default, to choose the destination partition; records sharing the same key are routed to the same partition, which is what provides per-entity ordering and enables log compaction.
Read full answerHide full answer
Applies toApache Kafka 3.x (KRaft or ZooKeeper-based clusters)
How Partitioning Uses Keys
The default partitioner hashes the key (MurmurHash2) and takes the result modulo the partition count, so a given key consistently maps to the same partition as long as the partition count does not change.
A null key bypasses hashing; modern clients use a sticky partitioner that batches records to one partition for a while, then rotates, improving throughput for unordered data.
You can supply a custom partitioner when routing rules need to differ from key hashing, for example to isolate a tenant or region.
Why Keys Matter
Ordering: Kafka guarantees order only within a partition, so keying by an entity ID (user, order, device) keeps that entity’s events in sequence.
Locality: a consumer processing a key sees the full history for that entity without cross-partition coordination.
Log compaction: compacted topics retain the latest record per key, so keys are mandatory for changelog-style topics and Kafka Streams state stores.
Stream processing: Kafka Streams joins, aggregations and windowed operations rely on keys to co-partition and group data correctly.
Key Design Trade-offs
Approach
Benefit
Risk
Stable high-cardinality key (user ID, order ID)
Even distribution, per-entity ordering
None significant
Low-cardinality key (country, status)
Simple grouping
Hot partitions and consumer lag
null key
Maximum throughput for unordered events
No ordering, no compaction
Composite key
Joins across multiple attributes
Larger keys, more hashing cost
Operational Considerations
Partition skew: a hot key can saturate one partition and its broker; monitor per-partition produce rate and consumer lag.
Partition count changes: adding partitions breaks the key-to-partition mapping, so plan partition counts before production traffic.
Serialization consistency: producers and consumers must agree on key serialization; a mismatch can create distinct keys or deserialization failures.
Compaction limits: a compact-only topic keeps the latest record for each key indefinitely, while older values for that key are removed over time. Tombstones (null values) are kept for delete.retention.ms and then removed; retention.ms/retention.bytes apply only when the topic uses cleanup.policy=delete or compact,delete.
Best Practices
Choose immutable, high-cardinality business identifiers as keys.
Avoid null keys when ordering or compaction is required.
Keep keys small; large keys increase network and storage overhead.
Test partition distribution with production-like key cardinality before go-live.
Alert on per-partition lag, not just consumer-group lag, to catch hot keys early.
18
What do the Kafka producer acks settings 0, 1 and all mean?
KnownRevise
The acks producer setting controls how many broker acknowledgements Kafka waits for before considering a send successful. It is the primary knob trading durability against latency and throughput.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x (KRaft), confluent-kafka clients
How Each Value Behaves
acks=0: The producer sends the record and immediately considers it successful without waiting for any broker response. This gives maximum throughput and lowest latency, but the broker may never have received the record; a broker failure or network drop can silently lose data. There is no delivery confirmation, so broker-side loss cannot be detected or recovered through acknowledgements.
acks=1: The producer waits for the partition leader to acknowledge after appending the record to its log. This protects against a single failed send being retried, but if the leader crashes before replicating to followers, the record can be lost even though the producer believed it succeeded. It was the default in older clients.
acks=all (also accepted as -1): The producer waits until the leader and all in-sync replicas (ISR) acknowledge. This gives the strongest durability guarantee: a record is not acknowledged until it exists on every replica currently in the ISR. Latency is higher because it waits for the slowest in-sync replica.
Comparison
Setting
Waits for
Durability
Latency
Typical use
0
Nothing
Lowest
Lowest
Metrics, logs where loss is acceptable
1
Leader only
Medium
Low
General workloads, default in older clients
all
Leader + all ISR
Highest
Highest
Financial, audit, critical events
Production Considerations
acks=all alone is not enough. Pair it with min.insync.replicas (broker/topic config) so the write fails rather than silently degrading when too few replicas are in sync. A common safe setup is replication factor 3 with min.insync.replicas=2: a write succeeds if at least 2 replicas (leader + one follower) acknowledge, and fails if only the leader is available.
Idempotence (enable.idempotence=true) should be enabled with acks=all to prevent duplicates on retries. Java clients 3.0+ default it on when acks=all; set it explicitly in other clients or versions.
Ordering: With retries enabled and idempotence off, a failed batch retried after a later batch succeeds can reorder messages. Idempotence plus max.in.flight.requests.per.connection=5 (or 1 without idempotence) preserves per-partition ordering.
Exactly-once semantics require acks=all, idempotence, and transactional producers/consumers (transactional.id, isolation.level=read_committed).
Monitoring: Watch record-error-rate, request-latency-avg, and ISR shrinkage. A shrinking ISR with acks=all increases latency and can cause producer timeouts; alert on UnderReplicatedPartitions.
Best Practices and Pitfalls
Do not use acks=0 for data you cannot afford to lose; there is no delivery confirmation at all.
Do not assume acks=1 is durable: a leader crash before replication loses acknowledged data.
Always set min.insync.replicas when using acks=all, otherwise a single surviving replica silently weakens the guarantee.
Avoid mixing acks=all with very low request.timeout.ms; producers will time out and retry, increasing load.
Prefer acks=all as the default for production topics; tune linger.ms and batch.size to recover throughput rather than weakening durability.
19
What does an idempotent Kafka producer guarantee?
KnownRevise
An idempotent Kafka producer prevents a retried message from being appended multiple times to the same Kafka partition. It uses a producer ID, epoch, and per-partition sequence number so the broker can recognize retries as duplicates rather than creating duplicate records.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x
How It Works
Producer ID (PID): The broker assigns the producer a unique identifier when it initializes the producer session.
Epoch: Each producer session receives an epoch, allowing the broker to reject state from an older or fenced producer session.
Sequence number: The producer maintains a sequence for each topic partition.
Broker deduplication: If a request is retried, the broker compares the PID, epoch, partition, and sequence number. A repeated sequence is acknowledged without appending another record.
Safe retries: The producer can retry failed requests without creating duplicates, improving resilience without changing the application’s processing model.
Idempotence is scoped to retries performed by the same producer session. It does not automatically deduplicate two separate calls to send(), nor does it prevent duplication after the producer restarts and receives a new PID.
Required Configuration
Setting
Recommended value
Reason
enable.idempotence
true
Enables PID- and sequence-based deduplication. It defaults to true in Kafka 3.x and 4.x, but explicit configuration is clearer.
acks
all
Required for an idempotent producer and prevents acknowledgement before the record is replicated to the current in-sync replicas.
max.in.flight.requests.per.connection
5 or less
Kafka rejects larger values when idempotence is enabled. The default is 5.
retries
Retain the default or configure explicitly
Retries provide availability during transient failures; idempotence makes those retries safe from duplicate appends.
For stronger durability, configure the topic’s min.insync.replicas appropriately. For example, acks=all with min.insync.replicas=2 prevents writes when fewer than two replicas are in sync, trading availability for consistency.
Guarantees and Limitations
Per-partition ordering: Kafka preserves ordering within a partition, not across partitions or topics.
Not global exactly-once: Idempotence removes duplicates caused by producer retries. It does not make database writes, HTTP requests, or other external side effects exactly once.
Not application deduplication: Sending the same business event twice creates two producer operations and therefore two records. Use a stable event ID or database uniqueness constraint when that must be prevented.
Consumer behavior: A consumer that processes a record and crashes before committing its offset can process it again. This is normal at-least-once processing.
Exactly-once processing: Use Kafka transactions with a transactional producer and configure consumers with isolation.level=read_committed. External systems still require coordinated idempotency or transaction handling.
Production Failure Modes
OutOfOrderSequenceException can occur when broker or client sequence state is corrupted or when requests for the same producer ID and epoch contain an unexpected sequence number.
InvalidProducerEpochException can occur when the broker no longer accepts the producer’s epoch, such as when a newer producer instance has fenced the older one.
A producer restart may establish a new PID; an application-level retry after that restart can still duplicate an event.
Using acks=all can increase latency and reduce write availability during leader elections or replica failures.
With transactional producers, a stable transactional.id lets Kafka fence an older producer instance and prevent concurrent transactional writes.
Operational Best Practices
Monitor producer record-error-rate and record-retry-rate, along with broker logs for producer-sequence and epoch errors.
Alert on under-replicated or offline partitions and on sustained ISR shrinkage.
Use TLS with SASL_SSL or another secure transport for production connections.
Keep clients and brokers on supported, compatible versions and upgrade brokers before relying on newer producer behavior.
Use transactional IDs and initTransactions() only when transaction semantics are required; ordinary idempotence does not require Kafka transactions.
20
How do Kafka producer retries cause duplicates or message reordering?
KnownRevise
Producer retries in Apache Kafka resend a batch when an acknowledgment is missing or a transient error occurs. Because the broker may have already persisted the original attempt, retries can create duplicate records, and with multiple in-flight batches they can also break per-partition ordering.
Read full answerHide full answer
Applies toApache Kafka 3.x (KRaft) with confluent-kafka / Java clients
How Retries Work
The producer sends a batch and waits for an acknowledgment according to acks (0, 1, or all).
On retriable errors (for example NOT_ENOUGH_REPLICAS, network timeouts) or a missing ack within delivery.timeout.ms, the client retries up to retries, spacing attempts by retry.backoff.ms.
If a non-retriable error occurs or the delivery timeout is exceeded, the send fails and the application must decide whether to retry at a higher level.
How Duplicates Arise
The broker may successfully append the record but the acknowledgment is lost on the network; the producer retries and the broker appends it again.
Application-level retries after a failed send() or flush() can also produce duplicates.
An idempotent producer prevents this by attaching a producer ID and per-partition sequence numbers; the broker deduplicates retries with the same sequence number.
How Reordering Arises
With max.in.flight.requests.per.connection greater than 1, batches can be in flight concurrently.
If batch 1 fails and is retried only after batch 2 succeeds, the final log order can become batch 2 then batch 1.
With idempotence enabled, Kafka preserves order for up to 5 in-flight requests. With idempotence disabled, set max.in.flight.requests.per.connection=1 to guarantee order, at a throughput cost.
Key Configuration
Setting
Effect
retries
Maximum number of send attempts
acks
Durability level of each write
enable.idempotence
Deduplication and ordering protection
max.in.flight.requests.per.connection
Concurrent in-flight batches per connection
delivery.timeout.ms
Total time budget for a send including retries
Production Guidance
Keep enable.idempotence=True (default in Kafka 3.x clients) for exactly-once-per-partition writes.
For end-to-end exactly-once across consume, process, and produce, combine idempotence with transactions.
Monitor producer metrics such as record-retry-total, record-error-total, and request latency.
Avoid acks=0 when ordering or durability matters, since there is no acknowledgment to trigger a retry.
Keep max.in.flight.requests.per.connection at or below 5 when idempotence is enabled.
21
What do batch.size and linger.ms control in Kafka?
KnownRevise
batch.size and linger.ms are producer-side settings that control how records are grouped into batches before being sent to a Kafka broker. Together they balance throughput against latency: batch.size caps how large a batch can grow, while linger.ms caps how long the producer waits for more records to fill that batch.
Read full answerHide full answer
Applies toApache Kafka 3.x Java producer (librdkafka-based clients such as confluent-kafka-python use the same ideas but different defaults: `batch.size=1000000` bytes and `linger.ms=5`; Kafka 4.0 also changed the Java default `linger.ms` to `5`)
How Batching Works
A Kafka producer does not send every record immediately. Instead, it accumulates records for the same partition into an in-memory batch. A batch is sent when any of these happens:
The batch reaches batch.size bytes.
linger.ms milliseconds have elapsed since the first record was added to the batch.
The producer’s send buffer is under memory pressure (buffer.memory).
flush() or close is called.
This means linger.ms = 0 (the default) makes the producer send as soon as a batch is available, favoring low latency, while a positive value lets small bursts coalesce into larger requests, improving throughput and compression efficiency.
The Two Settings
batch.size: Maximum size in bytes of a single batch per partition. Default is 16384 (16 KB). Records larger than this are sent in their own batch. Raising it helps when individual records are large or throughput is high, but it increases memory use per in-flight batch.
linger.ms: Maximum time in milliseconds to wait before sending a partially filled batch. Default is 0. Raising it (for example to 5–20 ms) increases batching and throughput at the cost of added end-to-end latency.
Trade-offs
Setting change
Throughput
Latency
Memory / CPU
Increase batch.size
Higher
Slightly higher
Higher per partition
Increase linger.ms
Higher
Higher (bounded by value)
Lower request rate
Both at defaults
Lower
Lowest
More requests
Production Guidance
Tune these together with compression.type (for example lz4 or zstd): larger batches compress better, so raising linger.ms often amplifies compression gains.
Do not set linger.ms so high that it exceeds your latency SLO; it is an upper bound, not a fixed delay.
Watch producer metrics such as record-send-rate, batch-size-avg, request-latency-avg, and buffer-available-bytes. If batch-size-avg is far below batch.size, batching is ineffective and linger.ms may be too low or traffic too sparse.
batch.size is per partition, so a topic with many partitions multiplies memory usage; size buffer.memory accordingly.
These settings do not change delivery semantics: with enable.idempotence=true and acks=all, retries remain safe regardless of batching.
Common Pitfalls
Setting linger.ms high on a low-traffic topic can add latency without improving batching because batches rarely fill.
Oversizing batch.size without increasing buffer.memory can cause BufferExhaustedException under load.
Assuming batch.size is a global limit; it is per partition, so total in-flight memory can be much larger.
22
How does message compression work in Kafka producers?
KnownRevise
Compression in Kafka producers happens at the batch level: the producer groups records destined for the same partition into a batch, compresses that entire batch with the configured codec, and sends the compressed bytes to the broker, which stores them as-is.
Read full answerHide full answer
Applies toApache Kafka 3.x (KRaft or ZooKeeper mode) with the Java/Python clients; configuration names are stable across 2.x–3.x.
How It Works
Records are accumulated per partition into a RecordBatch (controlled by batch.size and linger.ms).
When the batch is sent, the producer compresses the whole batch with the codec named by compression.type.
The broker does not decompress by default; it stores and replicates the compressed batch, and consumers decompress on fetch.
Because compression is per-batch, a single record or tiny batches gain almost nothing. Batching is what makes compression effective.
Compression Codecs
Codec
CPU cost
Ratio
Typical use
none
none
1x
Debugging, already-compressed payloads
gzip
high
best
Cold storage, low-throughput, high ratio needs
snappy
low
moderate
General balanced workloads
lz4
very low
moderate
High-throughput, latency-sensitive pipelines
zstd
low–moderate
best among fast codecs
Recommended for most modern workloads
zstd is usually the best trade-off on Kafka 2.1+ and is a recommended choice for most workloads, but the client default remains none.
Key Configuration
compression.type — codec name; default none in many clients, so set it explicitly.
linger.ms — wait time to fill a batch (default 0); raising it improves compression and throughput at the cost of latency.
compression.type on the broker (default producer) — if set to a specific codec, the broker recompresses incoming batches, which costs broker CPU and is rarely wanted.
Operational Considerations
CPU vs. network/disk: compression moves cost to producers and consumers; measure end-to-end latency, not just broker throughput.
Idempotence/transactions: compression composes fine with enable.idempotence=true and transactions; the compressed batch is still the unit of retry.
Mixed codecs: different producers can use different codecs on the same topic; brokers and consumers handle each batch independently.
Monitoring: watch record-send-rate, batch-size-avg, compression-rate-avg, and producer CPU. Low compression-rate-avg with high CPU usually means batches are too small.
Upgrades: all modern clients support all codecs; very old consumers may lack zstd/lz4 support, so verify consumer versions before switching.
Best Practices
Prefer zstd for new clusters; use lz4 when producer CPU is the bottleneck.
Raise linger.ms (5–20 ms) and batch.size together so compression has enough data to work on.
Keep payloads uncompressed at the application layer (e.g., don’t gzip JSON before producing) and let Kafka handle it.
Avoid forcing recompression on the broker; leave compression.type=producer.
Benchmark with realistic record sizes; tiny records may not benefit from any codec.
23
How does a Kafka consumer read and process topic records?
KnownRevise
A Kafka consumer is a client that fetches records from assigned topic partitions, processes them, and records its progress through consumer-group offsets. Consumers normally join a consumer group, allowing multiple instances to share partitions and scale horizontally.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode, including the Java client and bundled CLI tools
How a Consumer Reads Records
Kafka topics are partitioned logs stored on brokers. A partition leader maintains the authoritative log, while followers replicate it for broker-failure tolerance; KRaft controllers manage cluster metadata and leadership.
After subscribe, polling triggers metadata discovery, group coordination, and partition assignment. Subsequent polls return records in a ConsumerRecords batch from the consumer’s assigned partitions. The application processes those records and then decides whether and when to commit the corresponding group offsets.
Common consumer patterns include:
Java or Python clients: Application code calls poll(), processes the returned records, and manages commits.
Kafka Connect sink connectors: Consume records from Kafka topics and write them to external systems.
Kafka Streams applications: Consume input topics, maintain local state, and produce output topics.
Kafka preserves ordering only within a partition. Records from different partitions can be processed concurrently, and key-based ordering requires all records for that key to remain in one partition.
Groups, Assignments, and Offsets
A group.id identifies a consumer group. The group coordinator assigns each partition to at most one active member; members beyond the available partition count remain idle.
Adding, removing, or restarting a member causes a rebalance, during which partitions can move. The cooperative sticky assignor can reduce unnecessary movement compared with an eager rebalance. Changing a topic’s partition count can remap keyed records and disrupt continuity for a key, so partition counts should normally remain stable.
A committed offset identifies the next record that the group should consume, not the last record physically processed. If the group has no valid committed offset, auto.offset.reset selects a starting position, commonly earliest or latest.
If processing prevents the application from calling poll() within max.poll.interval.ms, the coordinator removes the consumer from the group and reassigns its partitions. Uncommitted records can then be delivered again.
Processing and Delivery Guarantees
A poll cycle consists of obtaining an assignment, fetching records, processing the returned batch, and committing offsets according to the required delivery guarantee.
Processing pattern
Result
Process, then commit
At least once; a crash before commit can cause duplicate processing
Commit, then process
At most once; a crash after commit can lose processing
Atomically publish output records and consumed offsets in one Kafka transaction
Exactly once for that Kafka read-process-write flow, when the required transaction settings are used
For at-least-once processing, set enable.auto.commit=false and commit only after successful processing. Handle commit failures and rebalances explicitly, and make business operations idempotent where possible because failures can still produce duplicates.
For Kafka exactly-once processing, the consumer must use isolation.level=read_committed. The producer must use a unique transactional.id, enable idempotence, use acks=all, and place output sends and sendOffsetsToTransaction calls in the same producer transaction. read_committed returns committed transactional records only up to the last stable offset. These guarantees do not cover arbitrary database or API side effects; those require idempotency, an outbox, or another coordination mechanism.
Operational Behavior and Best Practices
max.poll.records limits the size of one returned batch. Large batches increase processing latency and the risk of exceeding the poll interval.
Keep the interval between poll() calls below max.poll.interval.ms; session.timeout.ms instead controls failure detection through missed heartbeats.
Monitor consumer lag by partition, poll latency, commit failures, rebalance frequency, fetch errors, broker disk usage, and under-replicated partitions.
Use bounded retries and a dead-letter topic for poison records. Account for duplicate delivery and avoid retries that permanently block later partitions.
On compacted topics, cleanup eventually removes obsolete records for the same key, but it is not an immediate update mechanism; consumers can observe multiple versions before compaction. Ordinary delete-policy topics expire records through retention policies.
Replication is not a backup. Mirror important topics to another cluster or external storage, and periodically test restoration.
When using a Schema Registry, configure the matching deserializer and compatible schemas, and monitor incompatible schema IDs separately from lag.
Use TLS and SASL/SCRAM in production, restrict ACLs to required topic and group operations, and never store plaintext credentials in source control.
Test rolling broker and consumer upgrades with representative workloads, verify lag recovery and rebalance behavior, and maintain rollback and incident-recovery plans.
24
What is a Kafka consumer group?
KnownRevise
A Kafka consumer group is a logical set of consumers that share the same group.id and cooperate to consume subscribed topic partitions. Kafka assigns each subscribed partition to at most one member of a group at a time, while different groups consume the same records independently.
Group membership: Consumers configure the same group.id to join the same group. Each consumer may receive multiple partitions and may remain idle if there are insufficient partitions.
Coordination: The broker acting as the group coordinator manages membership, heartbeats, committed offsets, and rebalances. The elected group leader calculates partition assignments using a partition-assignment strategy.
Partition ownership: A partition is delivered to only one consumer within a group, preserving partition-level ordering for that group. Other consumer groups maintain separate offsets and receive the records independently.
Rebalancing: When a member joins, leaves, times out, or changes its subscription, Kafka revokes and reassigns partitions as necessary. A consumer that stops calling poll() can be removed after max.poll.interval.ms, even if its background heartbeat thread is healthy.
Configuration and Rebalancing
Property
Default
Purpose
group.id
Required
Identifier shared by members of one group
session.timeout.ms
45000
Time without a heartbeat before the member is considered failed
heartbeat.interval.ms
3000
Background heartbeat frequency
max.poll.interval.ms
300000
Maximum permitted delay between poll() calls
max.poll.records
500
Maximum records returned by one poll()
enable.auto.commit
true
Enables automatic offset commits
auto.offset.reset
latest
Initial position when no valid committed offset exists
partition.assignment.strategy
RangeAssignor, CooperativeStickyAssignor
Ordered list of supported assignment strategies
The leader uses the first configured assignment strategy supported by every member, so the default list normally selects RangeAssignor. CooperativeStickyAssignor supports incremental rebalancing and should be configured explicitly when its reduced partition revocation is desired.
Offsets and Delivery Semantics
A committed offset identifies the next record to consume and is stored in the internal __consumer_offsets topic.
Mode
Requirements and behavior
Automatic commit
The default client commits positions during poll() cycles. This does not atomically coordinate processing with commits, so it does not by itself guarantee end-to-end at-least-once or exactly-once behavior.
At-least-once
Disable automatic commits and commit after the business side effect succeeds. A crash before the commit causes the records to be processed again, so handlers must be idempotent.
At-most-once
Commit before processing. A failure after the commit but before processing loses the record from the consumer’s perspective.
Exactly-once
Use a Kafka transaction, a transactional producer with a stable transactional.id, isolation.level=read_committed, and sendOffsetsToTransaction before commitTransaction. This atomically commits output and input offsets within Kafka; it does not make an arbitrary external database transaction atomic.
Producer idempotence prevents duplicate Kafka writes from that producer but is not, by itself, consumer exactly-once processing.
Operational Behavior
Lag: Monitor per-partition consumer lag, poll latency, processing latency, rebalance frequency, and commit failures. A growing lag means consumers cannot keep up with production.
Failure handling: Broker failover may delay rebalances, while consumer crashes cause partition movement, duplicate processing, and temporary lag spikes.
Offset retention: The broker setting offsets.retention.minutes defaults to 10080 minutes, or seven days. This retention applies when the group has no active members; an expired group subsequently initializes from auto.offset.reset.
Static membership: A stable, unique group.instance.id can preserve assignment during a brief restart. It does not prevent all rebalances, and concurrently running members must not share the same value.
Security: Grant consumers READ access to the required topics and READ access to the consumer group. Restrict group and topic administration separately and use TLS or SASL for non-local connections.
Best Practices
Size each group against the partitions available for its subscribed topics; additional consumers remain idle.
Use cooperative sticky assignment when reducing rebalance disruption is more important than compatibility with older clients.
Keep KafkaConsumer operations on the consumer thread. For long processing, pause affected partitions or use controlled worker processes with explicit pause, offset, and commit coordination.
Increase max.poll.interval.ms only when processing legitimately requires it; do not use it to hide stalled applications.
Prefer manual commits after durable processing for at-least-once behavior, and design handlers to tolerate duplicates.
25
How does Kafka assign partitions to consumers in a consumer group?
KnownRevise
Kafka assigns each partition of a subscribed topic to exactly one consumer within a group, using a group coordinator broker and a pluggable partition assignment strategy. The assignment is recomputed whenever group membership changes, and the result is distributed to all members through the JoinGroup/SyncGroup protocol.
Read full answerHide full answer
Applies toApache Kafka 3.x (brokers and clients, KRaft or ZooKeeper mode)
How Assignment Works
A consumer group has one group coordinator broker, selected by hashing the group ID to a partition of the internal __consumer_offsets topic.
All members send heartbeats to the coordinator. When a consumer joins, leaves, or the partition count changes, the coordinator triggers a rebalance.
During rebalance, one member (the leader, elected by the coordinator) runs the configured PartitionAssignor and produces an assignment map; the coordinator distributes it via SyncGroup.
Each partition is owned by exactly one consumer in the group at any time, which is what gives per-partition ordering guarantees.
Assignment Strategies
Strategy
Behavior
Best For
RangeAssignor
Divides partitions of each topic into contiguous ranges per consumer
Single-topic groups; can cause imbalance with multiple topics
RoundRobinAssignor
Interleaves partitions across consumers
Multi-topic groups; better balance
StickyAssignor
Minimizes partition movement on rebalance while keeping balance
Reducing rebalance disruption
CooperativeStickyAssignor
Incremental rebalancing; only moves partitions that must change
Production workloads; avoids stop-the-world pauses
Set the strategy via partition.assignment.strategy on the consumer. In Kafka 3.x, CooperativeStickyAssignor is recommended for production workloads, while the client default remains RangeAssignor.
Eager vs Cooperative Rebalancing
Eager (classic): all consumers revoke all partitions, then rejoin. Causes a full stop-the-world pause and reprocessing risk.
Cooperative (incremental): consumers only revoke partitions that are moving. Other partitions keep being consumed, reducing downtime. Requires all members to use a cooperative assignor.
Static Membership
Set group.instance.id to a stable value per consumer instance. On restart, the consumer rejoins with the same identity and can reclaim its previous partitions without triggering a full rebalance, which is valuable for rolling deploys and pod restarts in Kubernetes.
Guarantees
Ordering is guaranteed only within a partition, not across partitions of a topic.
A partition is consumed by one consumer at a time, so offsets advance monotonically per partition.
With CooperativeStickyAssignor, partitions that do not move are never revoked, preserving in-flight processing.
Operational Considerations
Rebalance storms: frequent consumer restarts or slow processing cause repeated rebalances. Monitor consumer group state and rebalance rate.
Consumer lag: track records-lag-max per partition; uneven assignment shows up as skewed lag.
Session timeouts:session.timeout.ms and max.poll.interval.ms must exceed worst-case processing time or the consumer is kicked out, triggering rebalance.
Partition count changes (adding partitions) force a rebalance and break key-based ordering assumptions for new partitions.
Best Practices
Prefer CooperativeStickyAssignor for production.
Use static membership (group.instance.id) for stateful or long-running consumers.
Keep max.poll.records sized so processing fits within max.poll.interval.ms.
Avoid mixing assignor types within one group.
Monitor per-partition lag, not just group-level lag, to detect skew.
26
What causes a Kafka consumer rebalance and why is it disruptive?
KnownRevise
A consumer rebalance occurs when Kafka recalculates membership or partition assignments for a consumer group. It is disruptive because affected consumers must pause or stop work while assignments change, and uncommitted records may subsequently be processed again.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.0
What Causes a Rebalance?
Kafka’s group coordinator manages each group.id. Common triggers include:
A consumer joining or leaving the group.
A consumer crashing or failing heartbeat checks.
A consumer exceeding the classic protocol’s max.poll.interval.ms without calling poll().
The subscribed topic gaining or losing partitions.
A consumer changing its subscription.
A planned close() that sends a LeaveGroup request.
Protocol behavior differs:
Classic consumer group protocol: Membership depends on poll() calls and heartbeats. If a consumer does not poll within max.poll.interval.ms, the coordinator removes it and reassigns its partitions. Assignors include eager strategies, which revoke all partitions before reassignment, and cooperative strategies, which revoke only partitions that must move.
New consumer group protocol: Preview in Kafka 3.9 and enabled by default in Kafka 4.0. It uses heartbeat-based membership and server-side assignment, and it does not remove a member merely because it exceeded max.poll.interval.ms.
For the classic protocol, cooperative sticky assignment can be selected with partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor. The client must also use the classic protocol, such as with group.protocol=classic. Built-in assignors consider subscriptions, available partitions and members, and the selected strategy; they do not infer arbitrary application capacity.
Why Rebalancing Disrupts Processing
Processing pause: Eager rebalancing revokes every assignment, while cooperative rebalancing affects only moving partitions.
Partition handoff: The new owner must fetch from the correct offset, and the previous owner must stop processing revoked partitions.
Duplicate processing: If processing completes but the offset commit fails or ownership is lost first, the next owner processes those records again.
Uneven load: New members can receive partitions, while existing members wait or receive different assignments.
Extra coordination: Group membership, metadata, assignment, and offset state must remain consistent across consumers.
Rebalancing does not inherently lose records that have been successfully committed to Kafka. Its principal risk is replaying work when application processing and offset commits are not atomic.
Offsets, Ordering, and Delivery Semantics
Kafka guarantees record order within a partition, not across partitions. For at-least-once processing:
Poll records.
Process them successfully.
Commit the offset representing the next record to consume.
Ensure commits never advance beyond work that has actually completed.
A ConsumerRebalanceListener can stop work and attempt a final commit in onPartitionsRevoked. Revocation commits are best-effort and may fail because a rebalance is already in progress. In onPartitionsLost, ownership has already been lost, so the consumer must not assume that another commit is possible.
Producer idempotence does not eliminate consumer-side duplicates. Exactly-once processing requires a transactional design, such as Kafka Streams exactly-once semantics with appropriately configured brokers, transactional producers, source-offset commits, and read_committed consumers. External side effects still require idempotency or an outbox-style workflow.
Monitoring and Troubleshooting
Use kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processing to inspect membership, assignments, current offsets, and lag.
During an incident, correlate:
Rebalance rate and duration.
Consumer lag before and after membership changes.
Classic-protocol max.poll.interval.ms violations.
Heartbeat failures and network instability.
Long-running handlers or blocked calls between poll() calls.
Deployments, rolling restarts, topic changes, and autoscaling events.
In the classic protocol, keep max.poll.interval.ms comfortably above worst-case processing time and call poll() before that deadline. A worker pool can isolate expensive work, but it must preserve per-partition ordering and commit only offsets whose records have completed. pause() and resume() can control work flow, but the consumer must still poll regularly.
Production Practices
Prefer the new protocol on Kafka 4.0 when all clients support it; otherwise configure cooperative sticky assignment for the classic protocol.
Process records sequentially within a partition unless weaker ordering is acceptable.
Use manual commits after successful processing for predictable at-least-once behavior.
Handle both onPartitionsRevoked and onPartitionsLost, and design processing to tolerate replay.
Use rolling restarts and review autoscaling policies to limit simultaneous membership changes.
Treat occasional deployment-related rebalances as normal; frequent rebalances accompanied by lag spikes usually indicate slow processing, network instability, or unstable membership.
Remember that longer timeouts reduce false removals but delay recovery from genuinely stuck or dead consumers.
27
What is cooperative incremental rebalancing and static membership in Kafka?
KnownRevise
Cooperative (incremental) rebalancing and static membership are consumer-group features that reduce rebalance disruption: cooperative rebalancing moves only partitions that must change instead of stopping all consumers, while static membership lets a restarted consumer reclaim its previous assignment without an immediate rebalance.
Read full answerHide full answer
Applies toApache Kafka 3.x (static membership since 2.3, cooperative rebalancing since 2.4)
How Cooperative Rebalancing Works
In an eager rebalance, every consumer revokes all partitions, then receives a new assignment, causing a global processing pause.
In cooperative rebalancing, the assignor computes the difference between current and target assignments.
Consumers keep partitions that remain assigned to them and only revoke partitions that must move.
Reassignment may take multiple rebalance rounds, but unaffected partitions continue processing.
All members of the group must use a cooperative assignor, such as CooperativeStickyAssignor.
How Static Membership Works
A consumer sets a unique group.instance.id.
The broker treats that member as persistent rather than purely session-based.
If the consumer disconnects, its assignment is retained for session.timeout.ms.
If it rejoins with the same instance ID before that timeout expires, it can resume without triggering a rebalance.
If the timeout expires, the member is removed and a rebalance occurs.
Monitor consumer lag, rebalance frequency, group member count, and session timeouts.
Keep client and broker versions compatible with the chosen rebalance protocol.
Common Pitfalls
Mixing eager and cooperative assignors in the same group.
Reusing one group.instance.id across multiple running consumers: the coordinator fences the older member, so the two instances keep knocking each other out.
Setting session.timeout.ms too low for deployment or GC pauses.
Assuming static membership prevents all rebalances; it only delays removal until the session timeout.
28
How does Apache Kafka commit consumer group offsets?
KnownRevise
Kafka stores a committed offset for each topic-partition in a consumer group. The broker acting as the group coordinator writes that progress to __consumer_offsets, allowing the group to resume after restarts or rebalances.
Read full answerHide full answer
Applies toApache Kafka 4.x, Java 17+
How Offsets Are Stored
Consumer group: group.id identifies the group whose progress is stored.
Coordinator: Consumers send OffsetCommit requests to the broker coordinating their group.
Internal topic: The coordinator writes offsets to the replicated, log-compacted __consumer_offsets topic.
Offset semantics: A committed offset is the next position to consume in that partition; each partition has an independent offset.
Missing offsets: If no valid committed offset exists, auto.offset.reset selects the starting position. Its default is latest; earliest and none are also supported.
Ordering: Offsets do not change Kafka’s ordering guarantee, which exists only within a partition. Committing past unfinished work can skip that work.
Automatic and Manual Committing
Mode
Configuration
Behavior
Automatic
enable.auto.commit=true (default)
During poll(), the client periodically commits positions returned by an earlier poll.
Manual synchronous
enable.auto.commit=false and commitSync()
The call blocks until the commit succeeds or fails.
Manual asynchronous
enable.auto.commit=false and commitAsync()
The commit proceeds asynchronously and can invoke a callback.
For business processing, the usual flow is:
Poll a batch of records.
Complete the durable business operation.
Commit the consumer positions with commitSync().
Continue polling.
If the application fails before committing, the group eventually rebalances and resumes from its previous committed offsets, replaying the uncommitted records. This provides at-least-once processing when combined with manual commits; handlers should still be idempotent because a successful business update followed by a failed offset commit can cause a duplicate.
Asynchronous commits reduce latency, but the application must handle their callbacks and ensure that an in-flight commit is not lost during shutdown.
Exactly-Once Offset Handling
Kafka transactions can atomically publish output records and commit the corresponding consumed offsets:
Use an idempotent transactional producer with a unique, stable transactional.id for each logical producer.
Begin a transaction and produce the output records.
Call producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata()) with the consumed offsets.
Commit the transaction.
Use isolation.level=read_committed on consumers that must see only committed transactional records.
This does not make arbitrary external side effects exactly once. Database or HTTP operations outside Kafka still require idempotency, deduplication, or an outbox/inbox pattern. Kafka Streams supports its own exactly-once state-store and output guarantees with processing.guarantee=exactly_once; downstream consumers still need read_committed.
Operations and Failure Modes
Lag: kafka-consumer-groups.sh --describe compares committed offsets with partition log-end offsets.
Rebalances: A consumer that exceeds max.poll.interval.ms can lose its partitions. Continue polling during long work or pause assigned partitions while still meeting the poll interval.
Commit failures: Coordinator elections, broker failures, network faults, and authorization errors can fail commits. The client retries retriable failures, but fatal exceptions must terminate or alert the application.
Retention: offsets.retention.minutes controls how long offsets for inactive groups are retained. Expired offsets do not delete topic records; the reset strategy determines where consumption resumes.
Replication: Monitor __consumer_offsets, under-replicated partitions, and broker health. Replication is not a backup; use an appropriate Kafka backup or replication strategy for business topics.
Security: Use TLS and SASL in production. Grant READ on the consumer group and READ and DESCRIBE on the input topic; Kafka group ACLs do not use a DESCRIBE group operation.
Code Example
Manual synchronous committing with the Java client:
import java.time.Duration;
import java.util.List;
import java.util.Map;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.serialization.StringDeserializer;
public class OrderConsumer {
public static void main(String[] args) {
Map<String, Object> config = Map.of(
ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
"kafka-1.example.com:9092",
ConsumerConfig.GROUP_ID_CONFIG, "order-processors",
ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false,
ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest",
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName(),
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
try (KafkaConsumer<String, String> consumer =
new KafkaConsumer<>(config)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofSeconds(1));
for (var record : records) {
processOrder(record.key(), record.value());
}
if (!records.isEmpty()) {
// Commits the next position for consumed assigned partitions.
consumer.commitSync();
}
}
}
}
private static void processOrder(String orderId, String order) {
// Replace with an idempotent, durable business operation.
}
}
What is the difference between auto commit and manual commit in Kafka?
KnownRevise
In Kafka, offset commits record how far a consumer group has read; auto commit lets the client persist offsets on a timer, while manual commit gives the application explicit control over when offsets are saved.
Read full answerHide full answer
Applies toApache Kafka 3.x (Java consumer client and confluent-kafka Python).
How Offset Commits Work
Each consumer tracks its read position per partition; committing writes that position to the internal __consumer_offsets topic.
With auto commit (enable.auto.commit=true), the client commits the offset of the last polled batch on a timer (auto.commit.interval.ms, default 5000 ms), typically during poll().
With manual commit, auto commit is disabled and the application calls commitSync()/commitAsync() (Java) or consumer.commit() (Python) only after it has finished processing.
Auto Commit
Advantage: minimal code and simple consumption loops.
Risk: the committed offset can move ahead of actual processing. A crash after commit but before processing causes at-most-once loss; a crash before commit causes at-least-once duplicates. Semantics are effectively mixed and hard to reason about.
Best suited for low-stakes telemetry where occasional loss or duplication is acceptable.
Manual Commit
Advantage: you commit only after successful processing, giving clear at-least-once semantics.
Trade-offs: more code and responsibility; commitSync() blocks and can stall the poll loop, while commitAsync() is non-blocking but needs error handling and retry on rebalance.
Exactly-once requires transactions: sendOffsetsToTransaction() plus a transactional producer and isolation.level=read_committed on consumers.
Comparison
Aspect
Auto commit
Manual commit
Control
Client-driven timer
Application-driven
Semantics
Mixed loss/duplicate risk
Clear at-least-once (or exactly-once with transactions)
Failure handling
Limited
Full control over retries and ordering
Complexity
Low
Higher
Production fit
Low-stakes streams
Critical pipelines
Best Practices and Pitfalls
Disable auto commit for any pipeline where correctness matters (enable.auto.commit=false).
Commit after processing completes, and handle onPartitionsRevoked to commit in-flight work before a rebalance.
Keep processing idempotent so at-least-once redelivery does not corrupt state.
Avoid committing per record; batch commits reduce overhead on __consumer_offsets.
For exactly-once, use transactional producers and read_committed consumers; never mix non-transactional writes with manual commits expecting exactly-once.
Monitor consumer lag and commit latency; stalled commits often indicate a blocked commitSync() or a slow downstream.
30
What happens when a Kafka consumer crashes before committing offsets?
KnownRevise
When a Kafka consumer crashes before committing offsets, the messages it already fetched but did not commit are redelivered to the replacement consumer after a rebalance, resulting in at-least-once processing and possible duplicate side effects.
Read full answerHide full answer
Applies toApache Kafka 3.x (KRaft mode)
Offset Commit Semantics
Kafka stores consumer progress as offsets in the internal __consumer_offsets topic. An offset is only durable progress once it is committed. Fetching a record does not advance the committed offset, so a crash before commit leaves the last committed position unchanged.
Crash Before Commit: What Happens
The consumer’s session expires after session.timeout.ms (or it leaves the group explicitly).
The group coordinator triggers a rebalance and assigns its partitions to surviving or new consumers.
The new owner starts reading from the last committed offset, so uncommitted records are consumed again.
Ordering within a partition is preserved, but the same record may be processed multiple times.
Duplicate Processing and Idempotency
Because redelivery is expected, downstream writes must be idempotent. Common patterns include: - Upserts keyed by a business key or Kafka record key. - Deduplication using a processed-event store keyed by topic-partition-offset. - Transactional outbox or database transactions that include the offset.
Auto-Commit vs Manual Commit
Auto-commit (enable.auto.commit=true) commits periodically based on auto.commit.interval.ms; a crash can still lose up to that interval of progress and cause duplicates.
Manual commit gives control over when progress is recorded, typically after successful processing.
Committing before processing gives at-most-once semantics and risks data loss; committing after processing gives at-least-once semantics.
Exactly-Once Considerations
Kafka’s exactly-once semantics apply to consume-transform-produce pipelines using idempotent producers and transactions. A consumer alone cannot make arbitrary external side effects exactly-once; idempotent application logic is still required.
Operational Considerations
Monitor consumer lag and rebalance frequency; frequent rebalances can indicate slow processing or unstable consumers.
Set max.poll.interval.ms high enough for worst-case processing time to avoid false rebalances.
Use a dead-letter topic for poison-pill records so one bad message does not block the partition.
Ensure graceful shutdown calls consumer.close() to commit final offsets and leave the group cleanly.
Best Practices
Prefer manual commits after successful processing for critical pipelines.
Make consumers idempotent and track processed offsets where side effects are not naturally idempotent.
Keep processing time below max.poll.interval.ms or use a separate worker thread with care.
Alert on rising lag, repeated rebalances, and commit errors.
31
How does Apache Kafka consumer lag work, and how is it monitored?
KnownRevise
Consumer lag is the difference between a consumer group’s committed offset and the end offset of a Kafka topic-partition. Monitoring it shows whether consumers are keeping up with producers and helps identify stalled consumers, slow processing, insufficient parallelism, and unevenly distributed work.
Read full answerHide full answer
Applies toApache Kafka 4.0.x–4.1.x with kafka-exporter 1.x
How Lag Is Calculated
For each topic-partition for which the group has a valid committed offset:
lag = log-end-offset - committed-offset
Log-end-offset: The next offset Kafka will reserve for an appended record or batch.
Committed-offset: The next offset the group intends to consume.
Total group lag: The sum of lag across the group’s partitions with valid committed offsets, normally its assigned partitions.
This is an offset difference, not an exact record count. Retention and compaction can remove records while leaving offset gaps; transactional control batches can also consume offsets without producing ordinary application records. Consequently, lag may overstate the number of records still available for processing. Track record age or a source-timestamp metric alongside lag to understand urgency.
A zero value means the numeric offset positions match, but it does not prove that every relevant business record was processed if offsets were skipped or records were removed. If no valid committed offset exists, treat the lag as unavailable rather than zero.
Inspecting and Monitoring Lag
The Kafka CLI command kafka-consumer-groups.sh --bootstrap-server kafka-1.example.com:9092 --describe --group order-processing reports CURRENT-OFFSET, LOG-END-OFFSET, and LAG for each partition. Use --members to inspect assignments and members, and --state to distinguish an empty group from one with no valid offsets.
The kafka-exporter project exposes:
kafka_consumergroup_lag
kafka_consumergroup_current_offset
kafka_consumergroup_log_end_offset
Prometheus queries such as sum by (consumergroup) (kafka_consumergroup_lag) show total group lag, while max by (consumergroup, topic, partition) (kafka_consumergroup_lag) retains partition-level detail. These metrics do not directly provide the age of the oldest unprocessed record; that requires application or additional telemetry.
Broker JMX metrics remain necessary for CPU, network, disk, ISR changes, request latency, fetch rates, and under-replicated partitions, but broker metrics alone do not provide reliable lag for every consumer group.
Production Failure Modes
Consumer crash: Lag continues growing while production continues. A replacement consumer triggers a rebalance, and lag decreases only after processing catches up.
Slow downstream processing: Lag grows despite healthy brokers and an active consumer.
Hot partition: One partition receives disproportionate traffic or requires expensive processing. Aggregate lag can conceal this even when it falls elsewhere in the group.
Producer burst: Lag may temporarily increase and then recover if the consumer has sufficient capacity.
Offset expiration: After a group has no active members for offsets.retention.minutes, its committed offsets expire. When a consumer subsequently joins without a valid offset, auto.offset.reset determines the initial position.
Retention or compaction gaps: Numeric lag can include offset positions whose records no longer exist or are no longer deliverable.
Unsafe offset commits: With enable.auto.commit=true, Kafka can commit a polled position before processing finishes, creating a risk of lost work. Disable auto-commit and commit after successful processing for at-least-once behavior; duplicate external side effects remain possible.
Kafka read-process-write exactly-once processing additionally requires an idempotent transactional producer with a stable transactional.id, consumers using isolation.level=read_committed, and one transaction containing both output records and the consumed offsets through sendOffsetsToTransaction. This does not make arbitrary database or API writes exactly once; those require idempotency or a transactional outbox pattern.
Operational and Security Practices
Alert on sustained lag growth rate and duration, not only absolute lag.
Monitor every partition to expose hot or unevenly processed partitions.
Correlate lag with rebalances, processing latency, consumer errors, and broker metrics.
Track backlog age or source-timestamp freshness when timely processing is an SLO.
Protect kafka-exporter with TLS and SASL, restrict its Kafka principal to the required group and topic permissions, and secure its HTTP metrics endpoint.
Never reset offsets automatically merely because lag is high; first determine whether the cause is retention, a failed consumer, uneven assignments, or insufficient downstream capacity.
32
What do auto.offset.reset earliest and latest mean in Kafka?
KnownRevise
auto.offset.reset is the consumer-side fallback used when Kafka finds no valid committed offset for an assigned partition. earliest starts at the partition’s log start offset, while latest starts at its log end offset and normally waits for subsequently published records.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x
Exact Behavior
The reset policy applies when:
A consumer group subscribes for the first time and has no committed offset.
A committed offset has expired or is otherwise unavailable.
The committed offset is outside the partition’s valid range, such as after retention removed the referenced records.
If a valid committed offset exists, Kafka resumes from that offset regardless of auto.offset.reset.
Situation
earliest
latest
New consumer group
Partition’s log start offset
Partition’s log end offset
Committed offset expired
Partition’s log start offset
Partition’s log end offset
Committed offset below retention boundary
Partition’s log start offset
Partition’s log end offset
Committed offset above partition end
Partition’s log start offset
Partition’s log end offset
Valid committed offset
Resume from committed offset
Resume from committed offset
The log end offset is the offset that will be assigned to the next record. It is not the same as the high watermark, which is the last fully replicated or stable offset and can be lower during replication lag. Consequently, latest does not mean “read the last existing message”; it normally causes the consumer to wait for records published after its reset position is established.
Consumer Groups and Delivery Semantics
The policy is evaluated independently for each partition assigned to a consumer. Kafka preserves record order within a partition; changing the reset policy changes only the starting position and does not reorder records.
auto.offset.reset does not provide a delivery guarantee by itself:
At least once: Process records before committing their offsets. A crash can cause records to be replayed.
At most once: Commit offsets before processing. A crash can cause records to be skipped.
Exactly once: Perform the read-process-write operation as one Kafka transaction, including the output records and consumer offsets in the transaction. Use a transactional producer and isolation.level=read_committed on consumers that must not observe aborted transactional records; read_committed alone does not make processing exactly once.
Enabling automatic offset commits does not by itself provide exactly-once processing and may commit a position before the corresponding business operation completes.
Configuration
Set auto.offset.reset on each consumer. The Java client and Python confluent-kafka client both use the property name auto.offset.reset with the values earliest and latest.
For command-line testing, kafka-console-consumer.sh accepts the setting through --consumer-property auto.offset.reset=earliest or --consumer-property auto.offset.reset=latest. A broker’s server.properties does not provide a default for this consumer-specific setting.
The policy only takes effect when no valid committed offset exists. Using a group that already has committed offsets continues from those offsets even if the configured policy is earliest.
Production Trade-offs
earliest: Useful for replaying retained history, rebuilding projections, or auditing events. It can create substantial consumer lag and broker load.
latest: Useful for services interested only in new events. It avoids replaying old data, but records published before the reset position is established are skipped.
Consumption gap: latest can leave an application missing records when a new group starts or its previous offset becomes invalid. The records remain in Kafka; the loss is at the application-processing level.
Duplicate processing: When earliest resets a group whose prior progress was lost, previously processed records can be replayed. Downstream operations should therefore be idempotent unless transactional exactly-once processing is used.
Retention and compaction: Kafka can reset only to offsets Kafka still retains. For compacted topics, offsets may point to gaps because older records have been removed or compacted away.
Access control: Reset behavior does not bypass Kafka ACLs. The consumer still needs appropriate permissions to read the topic and join or use the consumer group.
Monitoring and Incident Checks
Use kafka-consumer-groups.sh with the --describe option to inspect each partition’s current offset, log end offset, and lag. During incidents, check:
Whether the group has a valid committed offset.
Whether topic retention or offsets.retention.minutes invalidated that offset.
Consumer lag and rebalance frequency.
Partition availability and under-replicated partitions.
Whether latest was unintentionally configured for a group that requires complete history.
33
How do you scale Kafka consumers, and what limits the parallelism?
KnownRevise
Kafka consumer parallelism is bounded by partition assignment: within one consumer group, each partition is assigned to at most one consumer at a time. The useful ceiling is therefore the number of partitions the group can consume, so consumers beyond that ceiling remain idle while a rebalance assigns them no work.
Read full answerHide full answer
Applies toApache Kafka 4.x (KRaft), Java 17+, and `kafka-clients` 4.x
Assignment and Limits
Partition ownership: A consumer can own multiple partitions, but two members of the same group cannot own the same partition simultaneously. Broker replication protects partition data; it does not multiply consumer-group parallelism.
Ordering: Kafka preserves record order within a partition, not across partitions. Ordering for a key therefore requires a stable key-to-partition mapping. Increasing a topic’s partition count can remap existing keys under the default partitioner and disrupt per-key processing order.
Independent groups: Each consumer group receives its own assignment, so separate billing and analytics groups can consume every partition in parallel.
Scaling Kafka Consumers
Add partitions to increase the assignment ceiling. Partition counts cannot be reduced.
Add consumer instances with the same group.id, up to the number of relevant partitions across their subscriptions.
Parallelize within a partition when necessary, using bounded executors. Preserve ordering for related keys and commit only after a contiguous prefix has completed.
Tune fetching and polling: max.poll.records, fetch.max.bytes, and max.partition.fetch.bytes affect batch size and memory, but a small max.poll.records value does not permanently cap throughput.
Protect downstream systems: Kafka consumption should not outrun database connections, HTTP limits, or processing capacity.
Limit
Operational effect
Mitigation
Relevant partitions
Hard ceiling on useful group members
Add partitions or use separate workloads
Hot partition or key
One partition becomes a latency bottleneck
Change key distribution or partitioning
max.poll.interval.ms
Long processing causes member eviction
Process faster, keep polling, or raise the limit
Assignment strategy
Eager assignors revoke all partitions before reassignment
Prefer CooperativeStickyAssignor
Offset granularity
Offsets are committed per partition, not per record
Commit the next offset after a contiguous prefix completes
Broker, network, or downstream capacity
Fetching or processing slows despite idle consumers
Add capacity, use quotas, batches, and backpressure
Delivery Semantics
At-least-once: Disable auto-commit and commit offsets after successful processing. A crash or rebalance can replay records, so handlers should be idempotent.
Exactly-once: For consume-transform-produce workflows, use a transactional producer with enable.idempotence=true and a transactional.id, and set consumers to isolation.level=read_committed. Output records and consumed offsets must participate in the transaction.
Kafka Streams uses processing.guarantee=exactly_once_v2 (the older exactly_once value was removed in Kafka 4.0). Kafka Connect source connectors can run exactly-once when the workers set exactly.once.source.support=enabled and the connector supports it (exactly.once.support=required); this is not a general guarantee for every connector type.
Code Example
# Inspect the partition count.
kafka-topics.sh --bootstrap-server kafka.example.com:9092 \
--describe --topic orders
# Increase the partition count; choose a value greater than the current count.
kafka-topics.sh --bootstrap-server kafka.example.com:9092 \
--alter --topic orders --partitions 24
# Inspect assignment and lag.
kafka-consumer-groups.sh --bootstrap-server kafka.example.com:9092 \
--describe --group billing-consumer
import java.time.Duration;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.clients.consumer.OffsetAndMetadata;
import org.apache.kafka.common.TopicPartition;
import org.apache.kafka.common.serialization.StringDeserializer;
public class BillingConsumer {
public static void main(String[] args) {
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG,
"kafka.example.com:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing-consumer");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
StringDeserializer.class.getName());
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
props.put(ConsumerConfig.MAX_POLL_RECORDS_CONFIG, 500);
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
"org.apache.kafka.clients.consumer.CooperativeStickyAssignor");
Map<TopicPartition, OffsetAndMetadata> pending = new HashMap<>();
try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
consumer.subscribe(List.of("orders"));
while (true) {
ConsumerRecords<String, String> records =
consumer.poll(Duration.ofMillis(500));
for (ConsumerRecord<String, String> record : records) {
process(record);
TopicPartition partition =
new TopicPartition(record.topic(), record.partition());
pending.put(partition,
new OffsetAndMetadata(record.offset() + 1L));
}
if (!pending.isEmpty()) {
consumer.commitSync(pending);
pending.clear();
}
}
}
}
static void process(ConsumerRecord<String, String> record) {
// Idempotent business logic.
}
}
Production Operations
Monitor per-partition lag, rebalance rate, poll latency, and commit failures. Alert on sustained lag growth rather than an isolated spike.
Rebalances occur when members join or leave, assignments change, or processing exceeds the relevant timeout. A paused consumer must continue calling poll().
Static membership uses a stable, unique group.instance.id per instance. It can preserve assignment across restarts, but duplicate IDs cause fencing and must be avoided.
Kafka Connect’s task count is capped by the connector’s tasks.max setting; the connector may create fewer tasks (for example, a sink gains nothing from more tasks than topic partitions), and worker capacity limits how many run well. Kafka Streams tasks derive from the topology and input partitions; num.stream.threads controls how many threads execute them.
Retention and compaction affect storage and replay behavior but not the consumer-group assignment ceiling. Schema Registry can add serialization and validation overhead but does not increase parallelism.
Use TLS plus SASL for client authentication and grant least-privilege topic and consumer-group ACLs. Never place credentials in source code or container images.
34
What do Kafka’s at-most-once, at-least-once, and exactly-once semantics guarantee?
KnownRevise
Kafka’s delivery semantics describe the trade-off between record loss and duplicate processing. Exactly-once processing is available within Kafka’s transactional boundaries, but it does not automatically make arbitrary external side effects exactly once.
Read full answerHide full answer
Applies toApache Kafka 4.0 clients and brokers (KRaft)
Guarantees
Semantic
What it guarantees
What can go wrong
At-most-once
A record is not deliberately processed more than once by the consumer. The consumer commits its offset before processing the record.
A failure after the offset commit but before processing causes data loss. A producer that does not retry can also lose records before they are durably stored.
At-least-once
Records are not intentionally discarded. The consumer processes the record before committing its offset.
A failure after processing but before the offset commit causes the record to be delivered again, so duplicate processing is possible.
Exactly-once
Kafka atomically commits the produced output records and the consumed offsets as one transaction. Aborted transactional records are not exposed to read_committed consumers.
The guarantee applies to Kafka records and offsets. External database writes, HTTP calls, and filesystem operations are not covered unless they participate in the same transactional system.
Producer and Consumer Responsibilities
Producer acknowledgements and retries: For at-most-once behavior, a typical configuration is acks=0 with retries=0; the producer accepts that a record can be lost without a broker acknowledgement. For at-least-once behavior, use durable acknowledgements such as acks=all, enable retries, and normally enable enable.idempotence=true to avoid duplicates caused by producer retries.
Offset commit ordering: At-most-once consumers commit before processing. At-least-once consumers commit after successful processing. Automatic commits are generally unsuitable when the application needs precise control over this ordering.
Idempotent processing: At-least-once consumers should make processing effectively idempotent using a business identifier, deduplication table, or equivalent mechanism. Kafka producer idempotence does not prevent a consumer from processing the same business event twice after a crash.
Ordering: Kafka preserves order within a partition, not across an entire topic. Related records that require ordering must use the same partition key.
Exactly-Once Requirements and Boundaries
A Kafka consume-transform-produce workflow needs a transactional producer, a stable and unique transactional.id, and explicit transaction handling. The producer writes output records and submits the consumed offsets with its transactional offset-commit API, then commits the transaction. The consumer must disable automatic offset commits and use isolation.level=read_committed.
read_committed consumers do not advance past the last stable offset while an earlier transaction remains open, and they do not expose aborted transactional records. A transaction timeout, producer failure, rebalance, or fencing error can cause the transaction to abort and the input records to be redelivered.
Kafka Streams can provide exactly-once processing for supported Kafka-to-Kafka topologies with processing.guarantee=exactly_once. Kafka Connect support is connector-specific and also requires the worker and connectors to be configured for transactional processing. Schema Registry does not provide delivery guarantees.
Operational Implications
Monitoring: Track consumer lag, the last stable offset versus the log end offset, producer errors, transaction aborts, commit failures, under-replicated partitions, and rebalance frequency.
Incident handling: Treat transaction aborts and consumer fencing as retryable workflow events. Ensure the application can safely reprocess records after a restart or coordinator change.
Retention and compaction: Retention limits replay time, while log compaction removes obsolete records. Neither changes the delivery guarantee or prevents an application from losing its offset or deduplication state.
Recovery: Replication protects against broker failure, but not accidental topic deletion or incorrect application writes. Mirror critical topics where appropriate and test restoration procedures.
Security: Use TLS, SASL, and least-privilege ACLs for producers, consumers, administrators, and transactional workflows.
Choosing a Semantic
Use at-most-once only when loss is acceptable and duplicates are unacceptable. Use at-least-once for most durable workflows, combined with idempotent processing. Use exactly-once when all relevant state is Kafka records or offsets and the operational cost of transactions is justified; use an outbox pattern or another coordinated mechanism when side effects occur outside Kafka.
35
How do Kafka transactions provide exactly-once processing?
KnownRevise
Kafka transactions provide exactly-once processing by atomically committing output records and consumer offsets to the same transaction. A failed transaction is aborted, so read_committed consumers never observe its records; a successful transaction exposes both outputs and offsets together.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode
How Exactly-Once Processing Works
A transactional producer uses a unique transactional.id to coordinate writes with Kafka’s transaction coordinator:
initTransactions() initializes the producer and fences any older producer using the same ID.
beginTransaction() starts a transaction.
The producer writes output records.
sendOffsetsToTransaction() adds the consumed input offsets to that transaction.
commitTransaction() atomically makes the outputs and offsets visible.
If processing fails, abortTransaction() discards the output and offset changes.
If the producer crashes before commit, Kafka eventually aborts the transaction after its timeout. Because the input offset was not committed, the input is processed again, but the previous uncommitted output remains invisible. This prevents duplicate visible Kafka records.
Consumers using isolation.level=read_committed stop at the last stable offset while a transaction is open and ignore records from aborted transactions.
Required Configuration
Component
Configuration
Purpose
Producer
enable.idempotence=true
Deduplicates retries within the producer session
Producer
acks=all
Requires all in-sync replicas to acknowledge writes
Producer
max.in.flight.requests.per.connection<=5
Preserves ordering with idempotence enabled
Producer
transactional.id=<unique-instance-id>
Identifies and fences a transactional producer
Consumer
isolation.level=read_committed
Excludes aborted and uncommitted records
Consumer
enable.auto.commit=false
Supports manually committing offsets with sendOffsetsToTransaction()
Kafka Streams
processing.guarantee=exactly_once_v2
Uses Kafka 4.x exactly-once processing
The producer’s transaction.timeout.ms must not exceed the broker’s transaction.max.timeout.ms. Transaction metadata is stored in the broker’s internal transaction-state topic, whose replication and minimum ISR settings should be configured like other critical internal topics.
Scope and Ecosystem Integration
Kafka Streams: exactly_once_v2 combines transactional output writes, offset management, and state-store changelog processing. It provides exactly-once behavior for Kafka topics and Kafka Streams state stores.
Kafka Connect: Exactly-once support depends on the individual source and sink connectors. Do not assume it is enabled merely because the cluster supports transactions; connectors must explicitly support the required semantics.
Schema Registry: It does not participate in the transaction. Schemas should still be registered and compatibility-checked before production writes. Schema incompatibility can cause records to fail independently of transaction handling.
External systems: Kafka transactions do not include databases, HTTP calls, or other side effects. Use idempotent APIs, deduplication, or an transactional outbox when the target cannot participate in the Kafka transaction.
Production Failure Modes
Producer fencing: Starting a new producer with an existing transactional.id prevents the previous instance from committing. Use a distinct ID for each concurrently running instance.
Timeout or crash: An open transaction eventually terminates. Monitor transaction timeout, abort rate, producer errors, and consumer lag.
Long processing: Keep processing shorter than the transaction timeout and consumer poll interval.
Broker failures: Spread brokers across failure domains and size replication so a quorum remains available.
Partial visibility: Transactions provide atomic visibility in Kafka, not global ordering across partitions or topics.
Do not use read_uncommitted in applications that require exactly-once visibility.
Treat exactly-once as a guarantee within the Kafka boundary, not as a guarantee for arbitrary external side effects.
36
How does the transactional outbox pattern work with Kafka?
KnownRevise
The transactional outbox pattern decouples a database commit from event publication by writing business state and an event record to the database in one transaction. A relay later publishes pending outbox records to Kafka, providing at-least-once delivery in the normal success path.
Read full answerHide full answer
Applies toApache Kafka 4.x and PostgreSQL 14+; the pattern itself is version-independent
How It Works
The application starts a database transaction.
It updates the business table and inserts an event into an outbox table containing an event ID, aggregate key, topic, payload, creation time, and publication state.
It commits the transaction, making the business change and event atomic within PostgreSQL.
A relay claims pending rows and publishes them to Kafka, normally using the aggregate ID as the message key.
After Kafka acknowledges delivery, the relay marks the rows as published. Cleanup later removes or archives them according to policy.
If the relay crashes after Kafka acknowledges but before the database commit, the rows remain pending and are published again.
Kafka brokers store the topic’s partitions and replicas. Replication and acks=all provide Kafka-side durability, subject to the configured minimum in-sync replica count.
Events with the same key normally enter the same partition while the partition count remains unchanged. Kafka preserves producer order within that partition, but not across partitions or aggregates. Multiple relay instances must not publish different events for the same aggregate concurrently if ordering matters.
The outbox does not create an atomic transaction between PostgreSQL and Kafka. A Kafka transaction cannot make the database commit and Kafka publication one unit.
Delivery Guarantees
Approach
Failure behavior
Guarantee
Separate database and Kafka writes
Either write can succeed independently
Inconsistent state is possible
Transactional outbox relay
A crash after acknowledgement can cause republication
At-least-once delivery
Idempotent consumer
Duplicates may arrive, but a recorded event ID can suppress repeated business effects
Once-only business effect, not once-only delivery
Kafka transactions
Produced records and consumed offsets can commit atomically
Exactly-once Kafka read-process-write when fully configured
Producer idempotence prevents duplicates caused by retries within that producer session; it does not prevent the relay from publishing the same outbox row again after a crash.
Consumers should commit their offsets only after processing. For Kafka-to-Kafka read-process-write flows, exactly-once processing requires a transactional producer with a stable, instance-specific transactional.id, inclusion of consumed offsets in the transaction, and consumers using isolation.level=read_committed. External database side effects still require an idempotency strategy. Kafka Streams with processing.guarantee=exactly_once can protect Kafka state and output records, but not arbitrary external side effects.
Relay and Topic Design
Topics: Pre-create topics with enough partitions for throughput and ordering requirements. An example is three replicas with min.insync.replicas=2, requiring at least three suitable brokers.
Claiming rows: Use a short claim transaction or leased ownership fields. If using FOR UPDATE SKIP LOCKED, do not hold database locks during slow Kafka network calls, and preserve per-aggregate ordering.
Publishing: Enable idempotence and acks=all; also bound delivery waits and retry behavior so an unavailable Kafka cluster cannot stall relays indefinitely.
Schema validation: A schema registry can validate Avro, Protobuf, or JSON Schema payloads and enforce compatibility before publication.
Kafka Connect: A CDC connector can replace a custom relay when it captures only the outbox table and preserves the event ID and aggregate key. It does not remove the need for consumer idempotency.
Production Operations
Monitoring: Track oldest unpublished-event age, pending-row count, publish latency and errors, consumer lag, under-replicated partitions, and offline replicas.
Retention: Clean published outbox rows according to an audit or recovery policy. Kafka topic retention and outbox cleanup are independent.
Compaction: Compaction is intended mainly for keyed state or changelog topics. It is not a substitute for consumer deduplication because a duplicate may be delivered before compaction removes it.
Poison events: Quarantine malformed or permanently rejected events with their event IDs rather than blocking the entire relay indefinitely.
Recovery: Test database restores and Kafka disaster recovery. Replication is not a backup, and uncoordinated restores can produce gaps or duplicates.
Upgrades: Kafka 4.x uses KRaft-only metadata operation; do not mix ZooKeeper and KRaft configuration.
Security and Best Practices
Use TLS and SASL authentication for Kafka, restrict topics with ACLs, and grant the relay write access only to required topics.
Use separate topics for events with different confidentiality or ownership requirements; headers alone do not provide isolation.
Protect database credentials, use least-privilege access to the outbox table, and encrypt sensitive payloads before publication.
Validate schemas and payload contents at the relay boundary, and never place credentials or unnecessary personal data in Kafka headers.
Keep event IDs stable across retries and store processed event IDs in the same database transaction as the consumer’s business effect when exactly-once business processing is required.
37
How does Apache Kafka guarantee message ordering within a partition?
KnownRevise
Kafka guarantees partition-level ordering: records appended to the same partition have increasing offsets, and a consumer observes that partition in offset order. It does not guarantee relative ordering across partitions, because different partitions can be processed concurrently.
Read full answerHide full answer
Applies toApache Kafka 4.x brokers and Java client (`kafka-clients`) 4.x
Ordering Scope and Architecture
A topic contains one or more partitions. Each record receives an offset within its partition, and consumers read a partition’s records in increasing offset order.
Records in different partitions have no guaranteed relative order. For example, a record in partition 1 may be delivered before one in partition 0.
With the default partitioner, records with the same key normally enter the same partition while the partition count remains unchanged. Explicit partition selection gives producers direct control, while a hot key can concentrate load on one partition.
Replicas maintain the same log order. After a clean leader failover, committed records remain available; insufficient replication can lose unacknowledged records. Unclean leader election adds data-loss risk and should remain disabled for critical workloads.
Producer-Side Requirements
For reliable ordering with retries or broker failures:
Idempotence: enable.idempotence=true prevents duplicate writes caused by retries and preserves order for records sent to the same partition. It defaults to true in the Kafka 4.x Java client; explicitly setting it to false disables it, while enabling it requires acks=all and max.in.flight.requests.per.connection<=5.
Durability: acks=all waits for all current in-sync replicas to acknowledge the record. Configure min.insync.replicas to define the minimum acceptable ISR size.
In-flight requests: Keep max.in.flight.requests.per.connection no greater than 5 when idempotence is enabled.
Ordering context: Multiple producer instances can interleave writes into one partition, so application-level sequencing may still be necessary.
acks and replication settings primarily affect durability; they do not establish ordering between partitions.
Consumer-Side Requirements
Within a consumer group, each partition is assigned to one consumer at a time, and that consumer receives records in offset order. Concurrent processing can nevertheless complete application side effects out of order.
Process one record at a time where downstream ordering matters, or explicitly preserve sequence numbers when parallelizing.
Set enable.auto.commit=false when processing and offset commits must be coordinated.
For at-least-once processing, complete the side effect before committing its offset. A crash between those operations causes replay, so side effects must be idempotent or use an idempotency key.
A rebalance can move a partition to another consumer. Rebalance handling must therefore avoid overlapping processing and incomplete side effects.
Delivery Guarantees
Guarantee
Typical behavior
Ordering implication
At-most-once
Offset is committed before processing
Records can be lost, with minimal replay
At-least-once
Processing occurs before offset commit
Records can be duplicated; sequential processing preserves delivery order
Exactly-once within Kafka
A transaction commits output records and consumed offsets together
Aborted transactional records are excluded by read_committed consumers
Exactly-once processing requires a transactional producer with a transactional.id, coordinated offset submission such as the Java API sendOffsetsToTransaction, and commonly isolation.level=read_committed. read_committed excludes aborted transactional records but still returns committed and non-transactional records. It does not make an external HTTP request or database write exactly-once; use an idempotent side effect, transactional outbox, or equivalent integration.
Production Failure Modes
Retries without idempotence: Can create duplicates and, with multiple in-flight requests, reorder records relative to their original send order.
Concurrent consumer processing: Kafka’s read order remains intact, but application side effects can complete out of order.
Partition count: Too few partitions limits parallelism and concentrates hot keys; too many increase metadata, replication, and recovery overhead.
Under-replicated partitions: Can increase producer latency and errors, especially with acks=all and restrictive min.insync.replicas.
Consumer lag: A slow partition delays later records; monitor lag per partition.
Ordering requires serialization at some point: producers must preserve their send order for a partition, and consumers must preserve processing order where side effects depend on it. KRaft controllers manage cluster metadata but do not provide cross-partition ordering.
Retention and compaction: Retention removes old segments, while compaction keeps the latest value for a key subject to tombstone handling. Both can leave offset gaps, but surviving records remain ordered by offset.
Kafka Connect: Source and sink tasks can process partitions independently, so Connect does not create global ordering. Connectors requiring per-key sequence must preserve key affinity and destination ordering.
Kafka Streams: A task processes records from each input partition in order; repartition topics create new partition boundaries, and parallelism does not imply global ordering.
Schema Registry: Schema IDs and compatibility checks do not participate in broker sequencing.
Production deployments should combine these guarantees with TLS, SASL authentication, least-privilege topic ACLs, partition-level lag monitoring, under-replicated and offline partition alerts, producer error tracking, and tested rebalance handling. Test supported upgrade paths and client compatibility before rolling upgrades.
38
How do you handle poison messages with Kafka dead-letter topics?
KnownRevise
A poison message is a record that repeatedly fails processing because it is malformed, incompatible with a schema, or invalid under business rules. Route it to a dead-letter topic (DLT) after bounded retries, while ensuring the original source offset is committed only after the failure has been durably recorded.
Classify failures as transient or permanent. Retry transient errors with bounded exponential backoff; send permanent failures directly to the DLT.
Set enable.auto.commit=false and manually commit offsets only after successful processing or confirmed DLT publication.
If DLT publication fails, stop or pause the affected partition and do not commit its source offset. Kafka will redeliver the record after recovery or consumer restart.
Keep retry loops bounded. Blocking a partition for too long can trigger max.poll.interval.ms evictions and create a consumer-group rebalance storm.
Include the original topic, partition, offset, timestamp, error class, error message, and relevant headers in the DLT record.
DLT Design
A DLT is a normal replicated Kafka topic. Create it with the same partition count as the source topic so preserving the record key also preserves source-partition ordering:
At least once; a crash can create duplicate DLT records
Publish to DLT and commit the offset in one Kafka transaction
Exactly once for the Kafka DLT record and source offset
Use enable.idempotence=true and acks=all for the DLT producer. For transactions, configure a unique transactional.id per producer instance and consume the DLT with isolation.level=read_committed.
Ordering is guaranteed only within a partition. To retain source-partition order, process each partition serially, preserve the key, and use matching partition counts. Deduplicate nontransactional DLT records using the tuple (original-topic, original-partition, original-offset).
Kafka Streams provides DeserializationExceptionHandler; processing.guarantee=exactly_once_v2 makes Kafka output and offsets atomic, but does not include arbitrary external side effects.
Code Example
import sys
from confluent_kafka import Consumer, Producer, TopicPartition
class PoisonMessage(Exception):
pass
def process(value):
if not value:
raise PoisonMessage("empty or invalid payload")
bootstrap = sys.argv[1]
instance_id = sys.argv[2]
consumer = Consumer({
"bootstrap.servers": bootstrap,
"group.id": "payments-workers",
"enable.auto.commit": False,
"isolation.level": "read_committed",
})
producer = Producer({
"bootstrap.servers": bootstrap,
"transactional.id": f"payments-dlt-{instance_id}",
"enable.idempotence": True,
"acks": "all",
})
producer.init_transactions()
consumer.subscribe(["payments"])
delivery_errors = []
def delivered(err, _message):
if err is not None:
delivery_errors.append(err)
while True:
for message in consumer.consume(num_messages=100, timeout=1.0):
if message.error():
raise RuntimeError(message.error())
try:
process(message.value())
except PoisonMessage as error:
producer.begin_transaction()
try:
producer.produce(
"payments.dlt",
key=message.key(),
value=message.value(),
headers={
"original-topic": message.topic().encode(),
"original-partition": str(message.partition()).encode(),
"original-offset": str(message.offset()).encode(),
"error-class": type(error).__name__.encode(),
"error-message": str(error).encode(),
},
on_delivery=delivered,
)
producer.flush()
if delivery_errors:
raise RuntimeError(delivery_errors[0])
# Atomically commit the DLT record and source offset.
producer.send_offsets_to_transaction(
[TopicPartition(
message.topic(),
message.partition(),
message.offset() + 1,
)],
consumer.consumer_group_metadata(),
)
producer.commit_transaction()
except Exception:
producer.abort_transaction()
raise
else:
consumer.commit(
offsets=[TopicPartition(
message.topic(),
message.partition(),
message.offset() + 1,
)],
asynchronous=False,
)
Operations and Security
Monitor DLT production rate, DLT consumer lag, oldest unprocessed record age, retry rate, delivery failures, and source-consumer lag. Alert on sustained DLT growth or publication failures.
Set DLT retention according to investigation and replay requirements. Replication is not a backup; use appropriate Kafka backup, restore, and disaster-recovery procedures.
Restrict source-read, DLT-write, and DLT-read permissions separately. Use TLS and SASL, and restrict network access to approved infrastructure CIDRs such as 10.0.0.0/16.
DLT payloads and error headers may contain sensitive data. Minimize payloads, sanitize exception messages, encrypt topics where required, and apply retention or deletion policies.
Replay DLT records through a dedicated, validated workflow rather than blindly producing them back to the source topic.
39
How do you implement retries for failed Kafka message processing?
KnownRevise
Use bounded retries with retry topics and a dead-letter topic, and acknowledge the source offset only after processing or retry publication succeeds. For consume-process-produce flows, a Kafka transaction can atomically commit the produced record and source offset, but it cannot guarantee exactly-once effects in an external database or API.
Kafka brokers store partitioned, replicated logs; consumer groups divide partitions among consumers. Kafka has no consumer-side setting that automatically retries application processing errors, so the application must implement the retry policy.
Consume with enable.auto.commit=false.
Retry transient failures locally with exponential backoff and a strict attempt limit.
After exhausting local retries, publish the original payload to a retry topic.
Retry consumers process that topic, optionally delaying records until the time in an x-retry-at header.
Route permanent failures or exhausted retry attempts to a dead-letter topic.
Commit the source offset only after successful handling or successful publication to the retry/DLT topic.
Create retry and DLT topics with the same partition count as the source topic so the original partition can be preserved:
Commit before processing. A crash loses the message; normally unsuitable.
At-least-once
Process first and commit afterward. A crash can cause redelivery, so handlers must be idempotent.
Exactly-once within Kafka
Use isolation.level=read_committed plus a transactional producer for output records and consumed offsets.
Kafka guarantees ordering within a partition, not across partitions or topics. A failed record can allow later records from the same partition to succeed unless the application pauses that partition. Also keep processing and backoff below max.poll.interval.ms; otherwise the consumer is removed from the group and another consumer may process the same partition concurrently.
Producer enable.idempotence=true with acks=all prevents duplicates caused by producer retries. It does not make the consumer’s business handler idempotent. External side effects require a unique event ID, a database uniqueness constraint, or an outbox/inbox pattern.
Kafka Platform Integrations
Kafka Connect: Sink connectors support errors.retry.timeout, errors.retry.delay.max.ms, errors.tolerance, and errors.deadletterqueue.topic.name. Enable errors.deadletterqueue.context.headers.enable to retain source context.
Kafka Streams: Use processing.guarantee=exactly_once_v2 for Kafka state and output transactions. Application-level retry topics or error handlers are still needed for custom processing failures.
Schema Registry: Keep the original payload schema when copying records to retry and DLT topics. Store retry metadata in headers rather than changing the payload schema, and verify compatibility before deployment.
Code Example
This transactional Python consumer retries transient failures locally, then moves exhausted or permanent failures to retry/DLT topics:
import json
import time
from confluent_kafka import Consumer, KafkaError, KafkaException, Producer, TopicPartition
SOURCE_TOPIC = "orders"
RETRY_TOPIC = "orders.retry"
DLT_TOPIC = "orders.dlt"
MAX_LOCAL_ATTEMPTS = 3
class TransientError(Exception):
pass
class PermanentError(Exception):
pass
def process_event(raw):
try:
event = json.loads(raw)
except (json.JSONDecodeError, UnicodeDecodeError) as exc:
raise PermanentError("invalid JSON") from exc
if not event.get("order_id"):
raise PermanentError("missing order_id")
return event["order_id"]
def commit_or_route(consumer, producer, message, destination=None, error=None):
producer.begin_transaction()
try:
if destination:
headers = list(message.headers() or [])
headers.extend([
("x-source-topic", message.topic().encode()),
("x-source-partition", str(message.partition()).encode()),
("x-source-offset", str(message.offset()).encode()),
("x-retry-at", str(int(time.time() * 1000) + 60000).encode()),
("x-error", str(error or "processing failed").encode()[:512]),
])
producer.produce(
destination,
key=message.key(),
value=message.value(),
headers=headers,
partition=message.partition(),
)
producer.poll(0)
offsets = [
TopicPartition(message.topic(), message.partition(), message.offset() + 1)
]
producer.send_offsets_to_transaction(
offsets, consumer.consumer_group_metadata()
)
producer.commit_transaction()
except Exception:
producer.abort_transaction()
raise
consumer = Consumer({
"bootstrap.servers": "10.0.0.10:9092",
"group.id": "order-workers",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"isolation.level": "read_committed",
"max.poll.interval.ms": 300000,
})
producer = Producer({
"bootstrap.servers": "10.0.0.10:9092",
"transactional.id": "order-worker-1", # Unique among concurrently running workers
"acks": "all",
"transaction.timeout.ms": 60000,
})
producer.init_transactions()
consumer.subscribe([SOURCE_TOPIC])
try:
while True:
message = consumer.poll(1.0)
if message is None:
continue
if message.error():
if message.error().code() != KafkaError._PARTITION_EOF:
raise KafkaException(message.error())
continue
for attempt in range(1, MAX_LOCAL_ATTEMPTS + 1):
try:
process_event(message.value())
commit_or_route(consumer, producer, message)
break
except TransientError as exc:
if attempt == MAX_LOCAL_ATTEMPTS:
commit_or_route(
consumer, producer, message, RETRY_TOPIC, exc
)
break
time.sleep(0.1 * (2 ** (attempt - 1)))
except PermanentError as exc:
commit_or_route(consumer, producer, message, DLT_TOPIC, exc)
break
finally:
consumer.close()
The transactional.id must be unique among active producer instances. A transaction protects Kafka records and offsets, but an external API call that succeeds before a failed transaction can still be repeated.
Production Operations
Monitor source, retry, and DLT lag with kafka-consumer-groups.sh --describe; alert on retry growth, consumer rebalances, transaction timeouts, and growing DLT volume.
Pause or isolate a partition when strict ordering matters; otherwise later records may overtake a failed record.
Cap retry attempts and DLT retention to prevent infinite poison-message loops and unbounded storage.
Size brokers for retry spikes and configure transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2 on a production KRaft cluster.
Protect brokers and topics with TLS, SASL, and least-privilege ACLs. Redact sensitive payloads from error headers and logs.
Replay DLT records through a controlled tool that preserves the original event ID, audit trail, and duplicate protection.
40
How do Kafka’s time and size retention settings work?
KnownRevise
Retention determines how long Kafka retains records in a partition. Kafka supports time-based retention, size-based retention, or both, and deletion happens at whole-segment granularity rather than per record.
Read full answerHide full answer
Applies toApache Kafka 4.x (KRaft)
How Retention Works
Kafka stores each topic partition as an append-only log divided into segments. With the default cleanup.policy=delete, the log-cleaner thread periodically identifies segments that exceed the configured retention thresholds.
Retention is evaluated independently for each partition, not for the topic as a whole.
Kafka deletes complete segments, not individual records.
The active segment is never deleted. After it rolls, it can become eligible for deletion.
log.retention.check.interval.ms controls how frequently retention checks run and defaults to 300000 milliseconds.
Replicas perform cleanup independently, so disk usage can temporarily differ between replicas.
Time-Based Retention
Time-based retention uses the largest timestamp recorded in a segment. A rolled segment becomes eligible when its age exceeds the effective retention period.
Setting
Scope and behavior
log.retention.ms
Broker-level limit in milliseconds; defaults to -1.
log.retention.hours
Broker-level limit in hours; defaults to 168, or seven days.
retention.ms
Topic-level millisecond override.
retention.hours
Topic-level hour override.
A non-negative millisecond value takes precedence over an hour value. Otherwise, a non-negative hour value is converted to milliseconds; if neither applicable value is non-negative, time retention is unlimited. Topic-level values, when present, take precedence over inherited broker values.
Retention is approximate because it operates on rolled segments. For example, an active segment can remain beyond seven days if it has not rolled, while an older rolled segment may be deleted immediately once it exceeds the threshold.
Size-Based Retention
Size-based retention limits the log data retained per partition.
log.retention.bytes=-1 means unlimited broker-level size retention.
An explicit topic-level retention.bytes=-1 means unlimited size retention for that topic.
log.segment.bytes controls segment size and defaults to 1073741824 bytes, or 1 GiB.
Kafka deletes the oldest eligible segments until the partition is below the target where possible.
The active segment and segment-size granularity can cause actual usage to exceed the configured limit temporarily.
With cleanup.policy=delete, either the time or size threshold can independently trigger deletion.
Code Example
Inspect the effective topic configuration and then configure seven-day time retention plus a 10 GiB per-partition limit:
# Display the topic's retention configuration
bin/kafka-topics.sh \
--bootstrap-server broker.example.com:9093 \
--describe \
--topic orders \
--config
# Set time and size retention for the topic
bin/kafka-configs.sh \
--bootstrap-server broker.example.com:9093 \
--entity-type topics \
--entity-name orders \
--alter \
--add-config retention.ms=604800000,retention.bytes=10737418240
Capacity planning must multiply the per-partition limit by the partition count and replication factor, then allow for indexes, other topics, replicas, and deletion lag.
Operational and Failure Considerations
Consumer recovery: If a committed offset is older than the available log start offset, Kafka applies auto.offset.reset. Its default is latest; none causes the consumer to fail with an offset-out-of-range error.
Offset expiry: offsets.retention.minutes controls how long offsets for inactive consumer groups are retained and is separate from record retention.
Capacity pressure: Retention checks and file deletion are delayed operations, so disks require additional headroom.
Timestamp errors: Kafka defaults to producer CreateTime timestamps. A future timestamp can make a segment appear young and extend retention. LogAppendTime uses broker-generated timestamps instead.
Secure erasure: Kafka deletes segment files, but disks, SSDs, snapshots, or backups may retain recoverable bytes. Use encryption, restricted access, short retention, and appropriate storage erasure controls for sensitive data.
Monitoring: Track broker disk utilization, partition growth, log-start offsets, consumer lag, compaction and deletion metrics, and consumer-group rebalances.
Compaction Versus Deletion
cleanup.policy=compact retains the latest non-null record for each key rather than deleting data because of age or size. Tombstones represent deleted keys and are retained according to delete.retention.ms before the cleaner removes them.
Kafka also supports cleanup.policy=compact,delete. In that combined mode, a segment is deleted only when it is eligible under both compaction and deletion; satisfying only the time or size threshold is insufficient.
41
How does log compaction work in Apache Kafka, and when should it be used?
KnownRevise
Log compaction is Kafka’s mechanism for retaining the latest record for each key in a topic while removing records superseded by newer values with that key. It is useful for state-like data where intermediate values are obsolete, but it is not appropriate when every historical event must remain available.
Read full answerHide full answer
Applies toApache Kafka 3.x and 4.x
How Compaction Works
Kafka does not immediately remove an old value when a newer value with the same key arrives. The broker’s log cleaner periodically scans eligible log segments, identifies obsolete records, and rewrites the affected data into cleaned segments.
Key requirement: Records must have the same non-null key to supersede one another. Kafka routes a key to one partition, preserving its ordering within that partition.
Cleanup policy: A topic normally enables compaction with cleanup.policy=compact.
Asynchronous cleanup: Compaction does not immediately change what consumers observe. A consumer can still read an old value and its replacement before the cleaner removes the obsolete record.
Null keys: Records with a null key are not compacted because Kafka cannot associate them with a key whose latest value can be retained.
Offset gaps: Compaction removes records without renumbering the remaining offsets, so consumers will encounter gaps.
Offset preservation: Consumer-group offsets are independent of compaction. Compacting a topic does not reset committed consumer positions.
The cleaner generally does not compact the active segment. Settings such as min.cleanable.dirty.ratio, min.compaction.lag.ms, and segment.ms affect how quickly dirty segments become eligible for cleaning, but compaction remains asynchronous and should not be treated as immediate key-value storage.
Tombstones and Retention
A tombstone is a record with a non-null key and a null value. Consumers should interpret it as deletion of that key’s current value.
Kafka retains tombstones after compaction for the period defined by delete.retention.ms; it then removes the tombstone itself. Until the tombstone is removed, the key remains represented in the log. A topic can also combine compaction with a delete policy, in which case normal segment, time, or size-based retention can remove additional data.
When to Use It
Compaction is appropriate when only the latest state or metadata value matters:
Kafka Streams application state and compacted changelog topics
Latest-value configuration or feature-flag records
Rebuildable materialized views where intermediate updates are unnecessary
Long-lived deduplication markers, provided the application accounts for duplicates before compaction occurs
For example, repeatedly writing the latest customer address under key customer-123 allows Kafka to retain the newest address without preserving every previous address.
Suitable for compaction
Better without compaction
Latest state per key
Complete event history
Kafka Streams changelogs
Audit and compliance logs
Current metadata snapshots
Financial transaction ledgers
Rebuildable state
Data requiring historical reconstruction
Compaction should not be used as an archival deletion mechanism. It removes superseded values, but it does not provide precise deletion of every record associated with a subject, including data already retained in backups or in other partitions. Privacy workflows need an explicit deletion and retention design.
Guarantees and Non-Guarantees
Kafka provides ordering per partition, not global ordering across a topic. Compaction preserves that model because all records for one key share a partition.
Compaction itself provides neither at-least-once nor exactly-once processing guarantees:
Consumers can replay records after a restart or rebalance, including values that have already been processed.
Producer idempotence suppresses duplicate sends caused by retries within a producer session, but it does not prevent an application from deliberately publishing the same logical update multiple times.
Kafka transactions can provide atomic read-process-write behavior when the application is designed to use them.
Kafka Streams exactly-once processing is enabled with processing.guarantee=exactly_once; it requires a functioning broker transaction-state configuration and appropriately sized transaction-state settings.
Consumers of transactional topics that must not expose aborted transactions use isolation.level=read_committed.
These are application-processing guarantees, not guarantees created by compaction.
Production Best Practices
Use a reasonably small segment.ms or frequent segment rolls when faster compaction is important, while accounting for the additional segment and file-handle overhead.
Avoid an excessively aggressive min.cleanable.dirty.ratio; cleaning consumes broker CPU and disk I/O and can affect foreground traffic.
Monitor topic disk usage, log-cleaner metrics and backlog, compaction lag, under-replicated partitions, consumer lag, and tombstone handling.
Size partitions for expected key cardinality and throughput. Compaction reduces retained records but does not reduce the configured partition count.
Use TLS or SASL and topic-level ACLs where access control and encryption are required; compaction provides neither.
Use replication factor 3 across sufficiently independent brokers when the workload can tolerate that level of redundancy.
Test backups and restores carefully: a compacted backup represents retained state, not the topic’s complete historical event stream.
42
What are Kafka tombstone messages?
KnownRevise
A tombstone message is a Kafka record with a non-null key and a null value. In a compacted topic, it represents a deletion, allowing Kafka’s log cleaner to remove older records for that key while preserving the key’s deleted state.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.x
Purpose and Wire Format
A producer sends a tombstone by using a valid key and a null value:
Key: Identifies the record being deleted and must not be null.
Value: Is null, signaling deletion.
Offset: Advances normally like any other Kafka record.
Partition: Follows the producer’s configured partitioning behavior; an explicitly assigned partition takes precedence over key-based partitioning.
Kafka does not immediately remove a key from a topic. The tombstone is appended to the log, consumed by clients, and eventually handled by log compaction.
A null key is different: null-keyed records are excluded from compaction and do not act as deletion markers.
Compaction and Retention
Tombstones are primarily useful with topics configured for compaction:
cleanup.policy=compact: Compacts records and removes superseded values.
cleanup.policy=compact,delete: Compacts records and also applies normal time- or size-based log retention.
delete.retention.ms: Controls how long delete tombstones are retained; its default is 24 hours in Apache Kafka 3.9 and 4.x.
During compaction, Kafka keeps the latest non-null value for each key. When it encounters a tombstone, it can discard earlier values for that key. The tombstone remains temporarily so consumers have an opportunity to observe the deletion.
If cleanup.policy does not include compact, Kafka treats the null-valued record as an ordinary record. It is not compacted away as a deletion marker, although an application may still interpret its null value as a delete.
Consumer Behavior
A consumer receives the tombstone as a normal record whose deserialized value is null, provided its deserializer can represent null.
The consumer’s position advances when the record is polled. The committed consumer-group offset follows the consumer’s commit configuration and does not necessarily mean processing has completed; with auto-commit enabled, an offset can be committed before the corresponding record has been successfully processed.
Application behavior is important:
A compacted key-value consumer should treat null as “delete this key.”
A consumer that ignores null values may retain obsolete state in an external database or cache.
Consumers must support null deserialization; schemas and serializers must preserve the tombstone rather than rejecting or dropping it.
Kafka Streams uses tombstones in changelogs to signal removals from state stores.
Tombstones should be processed in the same ordered stream as normal updates. Ordering is guaranteed only within a partition.
Delivery and Failure Semantics
A tombstone has the same delivery guarantees as the producer configuration used to send it:
With enable.idempotence=true and acks=all, a producer avoids duplicate writes caused by retries.
At-least-once delivery can cause the same tombstone to be delivered more than once, so deletion operations should be idempotent.
Kafka’s transactional exactly-once model requires a transactional producer and consumers using isolation.level=read_committed; Kafka Streams additionally uses processing.guarantee=exactly_once.
A tombstone does not itself provide exactly-once delivery.
If a producer crashes before the broker acknowledges the record, the deletion may not have occurred. Monitoring producer errors and retry metrics is therefore as important as monitoring normal records.
Production Trade-offs and Failure Modes
Tombstones reduce storage for update-heavy compacted topics, but they introduce operational considerations:
Expired tombstones: If a consumer falls behind for longer than delete.retention.ms, it may miss the deletion marker. A later replay cannot rely on an expired tombstone to communicate that deletion and may need a current snapshot or changelog.
Ignoring nulls: Consumers that silently discard null values can maintain stale application state.
Compaction lag: Disk usage can remain high when the cleaner cannot keep up with write volume or segment configuration is unsuitable.
Schema compatibility: Avro, JSON Schema, and Protobuf consumers must be able to represent and process null values. Schema Registry rules should account for tombstone-compatible schemas.
Security: Protect compacted topics with appropriate Kafka ACLs, TLS, and authentication. A tombstone is a write to the topic, so write authorization is required.
Monitor consumer lag, tombstone processing failures, compaction throughput, segment size, dirty ratio, and disk utilization. During upgrades or incident recovery, verify that consumers can replay null values correctly and that state can be rebuilt safely from a current snapshot or the compacted log.
For example, configure compaction with kafka-configs.sh --bootstrap-server kafka.example.com:9092 --entity-type topics --entity-name customer-state --alter --add-config cleanup.policy=compact,delete, then verify it with kafka-configs.sh --bootstrap-server kafka.example.com:9092 --entity-type topics --entity-name customer-state --describe.
43
How do you choose partition counts for Kafka topics?
KnownRevise
Choose a topic’s partition count from expected throughput, required consumer concurrency, key distribution, and operational limits. Partitions are the unit of Kafka parallelism, and each partition is an independently replicated, ordered log.
Read full answerHide full answer
Applies toApache Kafka 4.x
How Partitions Affect the System
Producer parallelism: A producer can send records to multiple partitions concurrently. More partitions can increase aggregate throughput, but only if producers distribute data across them.
Consumer parallelism: Within one consumer group, at most one consumer receives records from a partition at a time. A group with more consumers than partitions leaves extra consumers idle.
Ordering: Kafka preserves order within a partition. For same-key ordering across producer retries, use an idempotent producer with enable.idempotence=true; otherwise, set max.in.flight.requests.per.connection=1. Partition ordering alone does not preserve producer send order in a non-idempotent retry scenario.
Replication: The replication factor is separate from partition count. For example, 12 partitions with replication factor 3 require 36 partition replicas across the cluster.
Failure handling: Each partition has one leader and follower replicas. A broker failure can affect several replicas hosted on it; leader election promotes an eligible in-sync follower. If the in-sync replica set falls below min.insync.replicas, production with acks=all can block.
Decision Criteria
Factor
How it affects partition count
Throughput
Benchmark realistic record size, compression, acknowledgements, and replication. Add partitions only if broker and network capacity remain sufficient.
Consumer concurrency
Use approximately the maximum number of active consumers required by each consumer group. Different groups can each consume every partition.
Ordering requirements
All records requiring strict mutual ordering must use the same partition, normally by using the same key.
Key cardinality
A few very hot keys can overload one partition regardless of the total partition count.
Retention and compaction
More partitions mean more log directories, segment files, leader elections, and replica operations.
Recovery time
Excessively large topics can lengthen reassignment and recovery operations because partitions move independently.
A useful starting estimate is the greater of the required consumer concurrency and the throughput-derived partition count. Validate it with production-like load tests rather than relying only on a records-per-second calculation.
Avoid creating thousands of nearly empty partitions for a low-volume topic. Metadata, file descriptors, replica scheduling, and reassignment work all grow with partition count. Conversely, one heavily loaded partition can become a throughput bottleneck.
Creating and Increasing Partitions
# Create a topic with 12 independently replicated partitions.
bin/kafka-topics.sh \
--bootstrap-server kafka-1.example.com:9093 \
--create \
--topic orders \
--partitions 12 \
--replication-factor 3 \
--config min.insync.replicas=2 \
--config retention.ms=604800000
# Increase the partition count when additional throughput or consumers are needed.
bin/kafka-topics.sh \
--bootstrap-server kafka-1.example.com:9093 \
--alter \
--topic orders \
--partitions 24
Kafka supports increasing a topic’s partition count, but not decreasing it. With Kafka’s default partitioner, increasing the count changes the hash-to-partition mapping. Existing records remain in their original partitions, while new records for an existing key can move to another partition, so ordering across the increase is not preserved. If that ordering matters, stop and flush producers before the alteration or migrate keyed data through a versioned topic; draining consumers alone cannot restore ordering.
Production Failure Modes
Hot partition: One key or partition receives disproportionate traffic. Inspect per-partition byte and request rates; distributing keys imperfectly may not fix an underlying hot-key design.
Under-replicated partitions: Insufficient brokers, failed disks, throttling, or slow replicas can leave the in-sync replica set below the configured size.
Consumer lag: More partitions do not help if the group already has more consumers than partitions or if processing is slower than ingestion.
Delivery semantics: acks=all waits for acknowledgment from all currently in-sync replicas, subject to min.insync.replicas, but does not provide exactly-once delivery by itself. At-least-once consumers should use enable.auto.commit=false and commit offsets after successful processing, while accepting that crashes can cause duplicates. Kafka exactly-once processing requires transactional producers, an atomic consume-transform-produce flow, and consumers using isolation.level=read_committed; it does not make arbitrary external side effects exactly once.
Rebalance pressure: Adding partitions can trigger consumer group rebalances. Use suitable session.timeout.ms and max.poll.interval.ms values and make processing idempotent.
Operational Practices
Start conservatively, benchmark, and increase partitions before individual brokers or network paths saturate.
Record partition count, replication factor, expected maximum consumers, and keying strategy in the topic’s runbook.
Keep replication factor independent from partition sizing. A common production baseline is replication factor 3 and min.insync.replicas=2, provided the cluster can tolerate losing one broker.
Apply topic-level ACLs, TLS, and SASL appropriately. Kafka authorization is normally managed at topic, group, and cluster-resource levels rather than independently per partition.
For compacted topics, validate tombstone handling and ensure consumers have reached the required end offset; deletion reduces storage but does not immediately remove every segment file.
44
How can the partition count of an Apache Kafka topic be reduced?
KnownRevise
No—partition count reduction is not supported for an existing Apache Kafka topic. The only supported in-place change to the partition count is increasing it; reducing it requires recreating the topic or migrating to a new topic with fewer partitions.
Read full answerHide full answer
Applies toApache Kafka 4.x, including KRaft mode
Why Reduction Is Unsupported
A topic’s partition count is part of its metadata and defines separate ordered logs. Reducing that count would require merging partition logs, rewriting partition and offset mappings, and reassigning consumer responsibilities. Kafka intentionally avoids this destructive, error-prone operation.
The usual alteration command only increases partitions:
Requesting fewer partitions than the topic currently has fails. kafka-reassign-partitions.sh cannot help either: it reassigns partition replicas between brokers but cannot reduce the number of partitions.
Migration Options
Approach
Data handling
Offset impact
Operational impact
Delete and recreate with the same name
Data must be backed up and restored or producers must republish it
Old offsets may point beyond restored records
Usually requires downtime
Create a topic with a new name
Backfill, dual-write, or stream records between topics
New topic has separate offsets
Preferred for controlled production migrations
Increase the partition count
No reduction occurs
Consumers rebalance
Supported natively, but partitioning and key distribution change
For production, creating a new topic is usually safer. Migrate records, verify completeness, switch consumers, and retain the old topic temporarily for rollback and audit. Kafka replication protects against broker failures but is not a backup by itself.
Code Example
Run these commands from the Kafka installation directory. The following assumes the original orders topic has 12 partitions and the replacement should have 3:
# Inspect the current partition count and configuration
bin/kafka-topics.sh \
--bootstrap-server kafka.example.com:9092 \
--describe \
--topic orders
# This cannot reduce an existing 12-partition topic to 3
bin/kafka-topics.sh \
--bootstrap-server kafka.example.com:9092 \
--alter \
--topic orders \
--partitions 3
# Create a replacement topic with fewer partitions
# This requires at least three Kafka brokers for replication factor 3
bin/kafka-topics.sh \
--bootstrap-server kafka.example.com:9092 \
--create \
--topic orders-v2 \
--partitions 3 \
--replication-factor 3 \
--config min.insync.replicas=2 \
--config cleanup.policy=delete
Migrate records with a controlled producer/consumer process or during a dual-write phase. After validating the new topic, update producers and consumers to use orders-v2.
After cutover and the required retention period, the old topic can be deleted if delete.topic.enable is enabled:
Topic deletion is asynchronous, so verify that the topic is gone through the monitoring system.
Production and Security Considerations
Consumer offsets: Group offsets are associated with a topic name and partition. Recreating the same topic name with fewer partitions can leave stale offsets, causing consumers to skip records or start at invalid positions. Prefer a new topic and consumer group, or deliberately reset offsets after restoration.
Ordering: Kafka guarantees ordering only within a partition. When several source partitions are merged, ensure records for the same entity are written to the new topic in their original order.
Consumer lag: A smaller partition count reduces consumer parallelism and increases per-partition processing load. Validate throughput and lag before cutover.
Replication: Maintain an appropriate replication factor and min.insync.replicas. Monitor UnderReplicatedPartitions, OfflinePartitionsCount, disk usage, request latency, and consumer lag.
Security: Grant READ, WRITE, and DESCRIBE on orders-v2 to the relevant clients, CREATE on CLUSTER to topic administrators, and DELETE on orders to principals responsible for cleanup. Grant consumer-group ACLs separately as applicable. Preserve TLS, SASL, quota, and topic-configuration policies.
Downstream systems: Update Kafka Connect source/sink topics settings, Kafka Streams topologies, schemas, and Schema Registry subjects. With TopicNameStrategy, a new topic normally requires a new subject name and compatibility decision.
KRaft: The restriction is unchanged in KRaft mode; controllers enforce the same topic metadata operations.
Best Practices
Prefer a new topic and controlled migration over deleting and recreating a production topic in place.
Record the original topic configuration, partition assignments, ACLs, quotas, schemas, and consumer positions before migration.
Use idempotent producers and idempotent or transactional downstream writes where appropriate to control duplicates.
Keep the source topic until record counts, checksums, key ordering, lag, and business reconciliation all pass.
Load-test the smaller partition count; reducing partitions can create hot partitions and remove the parallelism needed for peak throughput.
45
How does adding Kafka partitions affect message ordering by key?
KnownRevise
Adding partitions can break end-to-end ordering for a key because Kafka guarantees ordering only within a partition. Existing records remain in their original partitions, while the default producer partitioner may route new records for that key to a different partition after the partition count changes.
Read full answerHide full answer
Applies toApache Kafka brokers and Java clients 3.x–4.x (KRaft and ZooKeeper-based deployments in 3.x; KRaft-only in 4.x)
How Partition Selection Changes
With the default Java producer partitioner, a non-null key is assigned using the positive Murmur2 hash of its serialized bytes and the topic’s current partition count. Conceptually:
Increasing the count changes the modulus, so the same key may map to a different partition. Custom partitioners can implement different, potentially stable, mappings.
Situation
Result
The key remains in the same partition
Ordering for that key continues in that partition
The key maps to another partition
Historical and new records for the key are split across partitions
The two partitions are consumed by one group
The newer record may be processed first
The key is null
There is no key-order guarantee; the default sticky partitioner may distribute such records among partitions
Kafka does not move or rewrite existing records when partition count increases. Once a producer observes the updated metadata, it uses the new count; a producer with stale metadata can temporarily continue sending to the old mapping.
Consumer and Delivery Effects
A consumer group must assign the new partition, normally through a rebalance, and may process it concurrently with existing partitions.
Consumer offsets are tracked per TopicPartition; offsets from different partitions cannot establish relative ordering.
Two partitions can represent different time ranges, so a consumer can observe records for one key as A, C, B.
acks=all waits for the current in-sync replicas to acknowledge, but fault-tolerant durability also depends on the replication factor and min.insync.replicas. It never provides cross-partition ordering.
Idempotence prevents duplicates caused by retries within a producer session and preserves ordering within a partition when correctly configured. It requires enable.idempotence=true; compatible settings include acks=all, retries>0, and max.in.flight.requests.per.connection no greater than 5.
The usual consumer-group pattern of processing before committing offsets is at-least-once. Kafka’s exactly-once processing applies to Kafka consume-transform-produce flows using a transactional producer (transactional.id), enable.auto.commit=false, isolation.level=read_committed, and sendOffsetsToTransaction. It does not automatically cover external side effects.
If a key has been split across partitions and that split cannot be prevented, consumers need an application-level sequence number, version, or causal dependency to detect and handle reordering.
Production Operations
Plan the rebalance: New partitions change group assignments; eager rebalancing can temporarily move all assignments, while cooperative assignment can reduce unnecessary movement.
Check partition health: A new partition starts without historical records and may temporarily have leader-election or ISR convergence delays.
Monitor the change: Track under-replicated and offline partitions, producer latency, request rate, consumer lag, and rebalance duration.
Expect persistent skew: Hash remapping can redirect a popular key’s new traffic to one new partition indefinitely while its historical data remains in the old partition.
Avoid uncontrolled expansion: Draining the old partitions before expanding does not help on its own: after expansion a key can still map to a new partition, so its history stays split. If strict per-key continuity is required, use one of the approaches below instead of a plain partition increase.
Remember topic configuration: num.partitions controls the default partition count for newly auto-created topics; it does not increase the count of an existing topic.
Preserving Key Ordering
There is no broker-side operation that automatically merges the histories of a key’s old and new partitions. Practical approaches are:
Use stable explicit routing: A custom partitioner can map each key to a fixed physical partition, preserving existing assignments but limiting the parallelism gained from additional partitions.
Perform a controlled migration: Pause producers, create a destination topic with a compatible partition count and partitioning scheme, replay records in per-key order, wait for consumers to catch up, cut over, and then resume production.
Use application-level ordering: Include an idempotency key and monotonic sequence or version so consumers can deduplicate, buffer, and reject stale records.
Serialize affected keys: Route sensitive keys through an ordered processing mechanism instead of allowing independent Kafka partitions to process them concurrently.
46
What is the role of Kafka Connect in Apache Kafka?
KnownRevise
Kafka Connect is a framework for running source and sink connectors that move data between Kafka and external systems. Source connectors publish external data into Kafka, while sink connectors consume Kafka records and write them to databases, queues, search indexes, or other services.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.x
Core Role
Kafka Connect standardizes the operational work required to integrate systems with Kafka without building a custom deployment pipeline.
Component
Responsibility
Connector
Defines the integration logic and its configuration
Source connector
Reads from an external system and produces Kafka records
Sink connector
Consumes Kafka records and writes to an external system
Worker
Hosts connector tasks and manages their lifecycle
Task
Executes the actual connector work
REST API
Starts, stops, reconfigures, and inspects connectors and tasks
A connector may run as one task or multiple tasks. The tasks.max configuration controls the maximum number of tasks, allowing a sink connector to consume partitions in parallel.
Architecture and Processing
In distributed mode, several Kafka Connect workers form a cluster. They coordinate task ownership through Kafka group-coordination mechanisms and store connector configuration, offsets, and status in internal Kafka topics.
The default internal topic names are:
connect-configs: connector and task configuration
connect-offsets: source connector offsets
connect-status: connector and task status
These names are configurable through the corresponding worker properties. The topics contain operational state, are normally compacted, and should not be manually deleted during normal operation. Their replication factors should be chosen for the available brokers and the required durability.
A source connector reads from an external system and produces records to Kafka. Connector-defined source offsets, such as a database log position, are persisted in Kafka-backed offset storage so that a restarted task can resume from its last committed position. A sink connector consumes Kafka records as an ordinary consumer group (named connect-<connector-name> by default) and commits its offsets to __consumer_offsets.
Converters serialize keys and values between Kafka and the Connect runtime. Connect is separate from Kafka Streams, which is an application library for stateful stream processing. Schema Registry is also separate, although Connect converters can integrate with it.
A task may complete an external write but fail before committing its Kafka offset.
After a restart, the same record can be processed again.
Duplicate handling therefore depends on the connector and destination system.
Kafka’s ordering guarantee applies within a partition, not globally across a topic. Concurrent processing can reorder records across partitions, while records within each partition remain ordered.
Exactly-once processing is connector-specific rather than automatic. For source connectors (Kafka 3.3+), the workers must set exactly.once.source.support=enabled, and the connector can then require it with exactly.once.support=required if it supports exactly-once. Sink connectors have no such switch: they achieve exactly-once only if the connector writes idempotently or stores offsets atomically in the destination. An idempotent producer alone does not provide end-to-end exactly-once semantics, and an external database write is not automatically included in a Kafka transaction. Such writes require connector-specific idempotence or transaction integration.
Deployment Modes
connect-standalone.sh runs one worker process and stores offsets locally. It is suitable for development and small, single-process deployments.
connect-distributed.sh runs a scalable worker cluster and stores state in Kafka. Distributed mode is the normal production choice because it provides task redistribution, persistent state, centralized configuration, and REST-based management.
Production Operations
Monitoring: Enable JMX metrics, collect worker logs, and alert on failed tasks, repeated restarts, unhealthy internal topics, and consumer-group lag for sink workloads.
Failure handling: A failed task is normally restarted and may be reassigned to another worker. Retry limits, backoff policies, dead-letter queues, and recovery procedures are connector-specific.
Security: Use TLS and SASL for Kafka connections, apply ACLs to connector and internal topics, and restrict the REST API with network controls, authentication, or a secured reverse proxy.
Scaling: Increase tasks.max only when the connector supports parallel processing and the source or destination can handle the resulting concurrency. Scaling workers alone does not create more connector tasks.
Upgrades: Canary worker and plugin upgrades, verify connector compatibility, and monitor task reassignment. Keep exports of connector configurations and a recovery plan for internal topic data.
Operational pitfalls: Do not delete internal topics during incidents, assume global ordering, expose an unauthenticated REST listener publicly, or describe Connect as providing universal exactly-once behavior.
47
What are source and sink connectors in Kafka Connect?
KnownRevise
A source connector reads data from an external system and publishes it to Kafka, while a sink connector consumes Kafka records and writes them to an external system. Kafka Connect runs each connector as one or more tasks managed by Connect workers.
Read full answerHide full answer
Applies toApache Kafka 4.x (Kafka Connect)
Responsibilities and Data Flow
Connector
Data flow
Typical example
Source connector
External system → Kafka topic
PostgreSQL change data capture → Kafka
Sink connector
Kafka topic → External system
Kafka topic → OpenSearch
MirrorMaker 2, implemented as a source connector
Source Kafka cluster → target Kafka cluster
Cross-region topic replication
A source connector implementation commonly polls a database, broker, filesystem, or SaaS API. A sink connector invokes connector-specific writes such as inserts, updates, deletes, or service requests. Connect does not automatically make those external writes idempotent; the connector and destination must prevent harmful duplicates or provide reconciliation.
Kafka Streams is different: it is a stream-processing library embedded in an application, whereas Kafka Connect is a worker-based runtime for integrating Kafka with external systems.
Runtime and Delivery Guarantees
Kafka brokers host partitioned, replicated topic logs. Source connectors act as producers, while sink tasks consume assigned topic partitions as part of Kafka consumer groups.
Worker: Runs the Connect runtime, connector plugins, and tasks.
Connector: Defines the connector class and its configuration.
Task: Performs the actual work. tasks.max sets an upper bound rather than guaranteeing that many tasks; sink task count is also constrained by partition assignment.
Offset storage: Connect stores source positions and sink offsets in its internal offsets topic, __connect_offsets by default. The name is configurable with offset.storage.topic.
Converters: Convert keys and values between Kafka bytes and the connector’s data model. JSON, Avro, and Protobuf converters are available.
Schema Registry: Stores schemas and supports compatibility enforcement when used with compatible converters.
Connect normally provides at-least-once delivery. A source can publish a record and fail before its position is committed, causing republication. A sink can complete an external write and fail before committing its Kafka offset, causing the record to be processed again.
Configure every worker consistently with exactly.once.source.support=required for sources or exactly.once.sink.support=required for sinks.
The connector must implement the corresponding exactly-once contract and configure an idempotent transactional producer with an appropriate unique transactional ID.
Kafka transactions make Kafka-side records and offsets atomic. Consumers that must exclude aborted or uncommitted transactional records should use consumer.isolation.level=read_committed.
For sinks, a Kafka transaction does not automatically make an external-system write atomic with the offset commit. The connector or destination must also support transactional integration, idempotent writes, or reconciliation.
Kafka preserves ordering only within a partition. Sink tasks may process different partitions concurrently, so global ordering must not be assumed; producers should use a partition key when related records must remain ordered together.
Operational Configuration
tasks.max: Increase parallelism only when the connector supports concurrent task execution.
consumer.enable.auto.commit: Worker-scoped control for sink-consumer auto-commit behavior; do not use the bare enable.auto.commit name in worker configuration.
offset.flush.interval.ms: Controls how frequently the worker flushes connector offsets.
errors.tolerance: Determines whether failed records stop processing or are tolerated.
errors.deadletterqueue.topic.name: Selects the dead-letter topic when error tolerance permits rejected records to be skipped.
schema.registry.url: Points compatible converters to Schema Registry.
Credentials: Store database passwords, API tokens, and cloud credentials in an approved secrets mechanism rather than source control.
Production Operations
Monitoring: Export JMX metrics and alert on failed tasks, repeated retries, sink consumer-group lag, and dead-letter-topic growth. Check task state through GET /connectors/{name}/status.
Data handling: Use replicated topics in production. Apply time or size retention to ordinary event topics; use log compaction for keyed state or Kafka Streams changelog topics.
Failure isolation: Configure bounded retries and a dead-letter topic for poison records. Give the dead-letter topic appropriate retention, replication, and access controls.
Security: Use TLS and SASL between Connect and Kafka, least-privilege Kafka ACLs, encrypted connector-to-destination traffic, and rotated credentials supplied through a secrets manager.
Upgrades: Test connector plugins independently, verify Kafka and plugin compatibility, upgrade distributed workers in a rolling deployment, and redeploy connectors only after the workers are healthy.
Recovery: Export connector definitions and plugin configuration through controlled configuration management, back up relevant source systems and schemas, and replicate the internal offsets topic appropriately.
KRaft: Apache Kafka 4.x uses KRaft mode, and Connect workers connect through bootstrap.servers rather than ZooKeeper.
48
How does Debezium change data capture work with Kafka?
KnownRevise
Debezium extends change data capture by converting database-log changes into Kafka records. Its source connectors usually run in Kafka Connect, while Kafka Connect sinks, Kafka Streams applications, Schema Registry, and ordinary consumers process the resulting event stream.
Read full answerHide full answer
Applies toDebezium 3.x with Apache Kafka 3.8.x or 4.x; PostgreSQL 13+ for the PostgreSQL-specific flow
CDC Architecture and Flow
Database log: PostgreSQL records changes in its write-ahead log. Debezium uses the pgoutput logical-replication plugin and emits only transactions that commit.
Publication: The PostgreSQL publication identifies the captured tables. The connector can manage it according to publication.autocreate.mode, or use an existing publication named by publication.name.
Replication slot: The logical replication slot, configured with slot.name, preserves required WAL data until Debezium confirms consumption. A disconnected or failed connector can therefore cause WAL accumulation.
Snapshot and streaming: The PostgreSQL connector normally takes an initial snapshot and then continues with incremental changes. Inserts, updates, and deletes use op values c, u, and d; snapshot reads use r.
Event envelope: A change event contains fields such as before, after, source, op, and ts_ms, with transaction metadata when enabled.
Kafka topic: With topic.prefix=inventory and table public.products, the destination is inventory.public.products. Kafka stores each topic as partitioned, replicated logs; the default Debezium key uses the table identity and primary-key columns.
Debezium records source progress in Kafka Connect’s offset store. Downstream Kafka consumers use consumer groups and commit their own offsets independently.
Delivery and Ordering Guarantees
Mode
Behavior
Requirements
At-least-once
A restart can cause an already captured event to be published again
Idempotent consumers or deduplication
Exactly-once to Kafka
Debezium atomically publishes records and advances source offsets in a Kafka transaction
Connector support, a transactional distributed Connect worker, and consumers using isolation.level=read_committed
Exactly-once source behavior is optional and must be enabled with exactly.once.source.support=required. For a highly available Kafka cluster, configure transaction.state.log.replication.factor=3 and transaction.state.log.min.isr=2.
Kafka preserves order only within a partition. With the default primary-key partition key, successive changes for the same row normally remain ordered in one partition; no ordering is guaranteed across partitions or tables. Tables without primary keys require an explicit key strategy if row-level ordering matters.
Kafka transactions do not make writes to an external database atomic with Kafka. End-to-end exactly-once applies only when the downstream processing and offset updates participate in the transaction, such as supported Kafka Streams flows; external sinks still require idempotency, transactional CDC, or reconciliation.
Kafka Integration
Kafka Connect: Hosts Debezium source tasks and can host sink connectors that write the events to databases, search indexes, or other systems.
Kafka Streams: Consumes change topics for filtering, aggregation, enrichment, and materialized views.
Schema Registry: Can store Avro or JSON Schema schemas used by compatible converters. Standard JSON, Avro, and JSON Schema converters have different registry behavior, so converter selection must match the deployment.
Consumer groups: Partition assignment and consumer offsets are handled by Kafka, not by Debezium.
Production Practices and Failure Modes
Kafka topology: Run at least three brokers for a replicated production deployment, pre-create topics with replication factor 3, and normally set min.insync.replicas=2. Use KRaft for metadata management; it is required in Kafka 4.x. Do not depend on broker auto-topic creation.
Connect HA: Use multiple Connect workers and monitor /connectors/inventory-source/status. Monitor task failures, restarts, record counts, snapshot progress, and producer errors.
CDC lag: Track source lag using Connect connector metrics and the difference between PostgreSQL’s current WAL position and the slot’s confirmed_flush_lsn. kafka-consumer-groups.sh --describe is appropriate for downstream Kafka consumers, not for the database-to-Kafka source lag.
Slot health: Monitor active, confirmed_flush_lsn, wal_status, and safe_wal_size in pg_replication_slots. Do not delete slots during routine cleanup. An invalidated slot caused by lost WAL can require a new snapshot.
Retention: Retain CDC topics long enough for required replay. Log compaction is useful for latest-state consumers, but it can remove intermediate changes needed for auditing or replay.
Schema changes: Test DDL and schema evolution. Breaking changes can stop the connector or make existing events unreadable to consumers, so deploy compatible schemas and coordinated consumer changes.
Security and recovery: Use TLS and SASL where appropriate, restrict the Connect REST API, grant the connector only required database privileges, and inject secrets from a secret manager. Protect PostgreSQL with base backups and WAL archiving for point-in-time recovery; Kafka does not replace database backups. Test rolling Kafka and connector upgrades, including offset compatibility and snapshot behavior.
49
What is Schema Registry, and why use it in Kafka?
KnownRevise
Schema Registry is a centralized service for storing, validating, and retrieving data schemas used by Kafka producers and consumers. It enables governed schema evolution, preventing incompatible message formats from silently breaking event processing.
Read full answerHide full answer
Applies toConfluent Platform 8.0.x (Schema Registry 8.0.x) with Apache Kafka 3.8.x or 4.0.x
Core Responsibilities
Schema Registry manages schemas in formats such as Avro, JSON Schema, and Protobuf. Although commonly used with Apache Kafka, it is a separate Confluent project rather than a component built into Kafka brokers.
It provides:
Schema storage: Schemas are persisted in the internal _schemas Kafka topic.
Schema validation: New versions can be checked against compatibility rules before registration.
Schema discovery: Producers and consumers can retrieve schemas through the REST API.
Serialization support: Confluent serializers embed a schema ID in each message, allowing consumers to locate the correct schema.
Governance: Centralizing schemas gives platform teams one place to enforce contracts, ownership, and lifecycle policies.
The REST API is commonly structured around endpoints such as GET /subjects/{subject}/versions, GET /schemas/ids/{id}, and POST /subjects/{subject}/versions.
How It Works
A typical event-processing flow is:
A producer selects a subject, often using TopicNameStrategy or another naming strategy.
The serializer obtains the schema’s immutable ID from Schema Registry.
The message includes that ID alongside the serialized payload.
A consumer fetches the schema by ID, caches it locally, and deserializes the payload.
Schema Registry does not store every Kafka message. It stores only schema metadata, while Kafka stores the actual records.
Subject organization is important:
Subject: A logical grouping of schema versions, commonly one per topic or record type.
Version ID: Identifies a specific version within a subject.
Schema ID: Identifies the exact schema used to encode a message.
Compatibility policy: Determines whether a new subject version is acceptable.
A service should generally use one stable naming strategy. Changing strategies can create separate subjects for the same logical contract and fragment governance.
Compatibility Policies
Compatibility checks occur when a new schema version is registered.
Policy
Requirement
NONE
No compatibility validation
BACKWARD
A consumer using the new schema can read data written with the previous schema
BACKWARD_TRANSITIVE
The new schema is backward-compatible with every previous version
FORWARD
A consumer using the previous schema can read data written with the new schema
FORWARD_TRANSITIVE
The new schema is forward-compatible with every previous version
FULL
The new schema satisfies both backward and forward compatibility with the previous version
FULL_TRANSITIVE
Both guarantees apply across every previous version
For Avro, BACKWARD is a common starting point because adding a field with a default value can allow newer consumers to read older records. Compatibility can be configured globally and overridden for individual subjects through the /config/{subject} endpoint.
Production Operations
Run at least two Schema Registry instances behind a load balancer and use a replicated _schemas topic. Important server settings include:
kafkastore.bootstrap.servers: Kafka brokers used for schema storage.
kafkastore.topic.replication.factor: Replication factor, normally at least 3 across availability zones.
listeners: Schema Registry listener, such as http://0.0.0.0:8081 behind a TLS-terminating proxy.
All instances must use the same backing Kafka cluster and configuration. Monitor:
Registry request latency, error rate, and restart count.
Compatibility rejection rates.
_schemas partition leadership, under-replicated and offline partitions, partition size, and compaction health.
Broker disk capacity and request queue depth.
Producer and consumer schema-fetch failures.
Back up the _schemas topic, including its subject-level compatibility configuration, to an off-cluster location, and test restoration. Kafka replication protects against broker failure but is not a substitute for backups. During upgrades, canary the new version, verify client compatibility, and monitor registration and retrieval errors.
Security and Failure Modes
Expose Schema Registry only over TLS, require authentication, and authorize access to sensitive subjects. The connection from Schema Registry to Kafka should also use TLS and appropriate ACLs. Never embed credentials directly in schema.registry.url; use the Confluent client’s supported credential mechanism, such as basic.auth.credentials.source=USER_INFO.
Common failure modes include:
Registry outage: Producers needing a previously unseen schema may fail. Producers with an already cached schema ID may continue serializing; consumers can continue only if their client has cached the schema metadata required for deserialization.
Incompatible registration: The new schema is rejected, protecting existing consumers but potentially blocking deployment.
Incorrect subject naming: Equivalent schemas are split across multiple subjects, weakening compatibility enforcement.
Schema-ID loss: Losing _schemas data can make existing message IDs unresolvable.
False confidence: Schema compatibility checks structure, not business meaning. A schema can be technically compatible while containing invalid values.
Use Schema Registry when independently deployed producers and consumers need a shared, governed contract. For low-risk internal systems with stable formats, it may add operational overhead, but it is especially valuable for event-driven platforms, shared APIs, regulated data, and long-lived consumers.
50
Comparing Avro, Protobuf and JSON Schema for Kafka messages
KnownRevise
Avro, Protobuf and JSON Schema provide different trade-offs between schema enforcement, evolution, interoperability, payload size, and operational complexity. Avro has the strongest Kafka-specific ecosystem, Protobuf is efficient for typed contracts, and JSON Schema is flexible for externally governed or dynamic data.
Read full answerHide full answer
Applies toApache Kafka 4.x and Confluent Schema Registry 7.x when used independently
Format Comparison
Concern
Avro
Protobuf
JSON Schema
Representation
Binary, schema-driven
Binary, typed messages
Textual JSON with an external schema
Schema enforcement
Strong field-level contract
Strong typed contract
Rich validation rules
Kafka integration
Mature serializers, converters, and tooling
Supported through serializers and Schema Registry
Supported through validators and Schema Registry
Typical size
Compact binary
Usually compact binary
Largest because of JSON syntax and repeated field names
Evolution model
Reader/writer schema resolution
Field numbers and wire compatibility
Constraint- and property-based compatibility
Human readability
Low
Low
High
Best fit
Event streaming and data pipelines
High-throughput typed systems
Heterogeneous APIs and dynamic payloads
Avro represents records as named fields and types, with support for unions and defaults. Its reader/writer schema resolution provides explicit compatibility rules.
Protobuf encodes field numbers and wire types instead of field names. Compatibility depends on preserving field numbers and using wire-compatible changes.
JSON Schema describes and validates a JSON document. It is not itself a binary serialization format, and a JSON message can exist without any machine-enforced schema.
Kafka and Schema Registry
Kafka brokers do not understand these schemas. Serialization and validation occur in producers, consumers, Kafka Connect converters, or Kafka Streams applications.
Confluent Schema Registry stores versioned schemas under subjects such as orders-value and orders-key. Registry-aware serializers typically attach a schema ID to the payload; clients retrieve the corresponding schema and cache it locally.
Avro: The payload uses Avro binary encoding, while Schema Registry manages the writer schema and its ID.
Protobuf: The payload uses Protobuf wire encoding, while Schema Registry manages the schema definition and compatibility checks.
JSON Schema: The payload remains JSON, and a validator can enforce the registered schema before transmission or consumption.
Compatibility checking is format-specific. Avro has defined BACKWARD, FORWARD, and FULL reader/writer semantics. Protobuf checking focuses on field-level and wire compatibility. JSON Schema checking depends on the draft and the constraints being changed. A successful registry check does not prove that existing consumers preserve the intended business semantics.
Evolution and Delivery Guarantees
Avro evolution: Add a new field with a reader-side default when old readers must continue working. Removing fields, introducing incompatible unions, or making changes that readers cannot resolve requires coordinated reader updates.
Protobuf evolution: Never reuse a field number for a different meaning. Reserve removed field numbers and names, and assess any type change for wire and semantic compatibility.
JSON Schema evolution: Adding an optional property is normally compatible only if existing schemas permit additional properties. Tightening required, types, ranges, patterns, or size constraints can break existing producers and consumers.
Schema choice does not change Kafka’s delivery guarantees:
At-least-once: A consumer processes a record before committing its offset. A failure after processing but before the commit can cause redelivery and duplicates.
Exactly-once within Kafka: For consume-transform-produce workflows, use an idempotent transactional producer, keep transactional.id stable per logical producer, submit consumed offsets atomically with output records using send_offsets_to_transaction, and set consumers to isolation.level=read_committed. Kafka Streams can provide the equivalent transactional processing model. Arbitrary external side effects still require an idempotent or transactional design.
Ordering: Kafka preserves ordering within a partition, not across a topic. Consumers must also be able to deserialize each format’s schema evolution, while application-level semantic compatibility remains a separate concern.
Production Trade-offs
Performance: Protobuf is often attractive for small numeric messages, while Avro is competitive and has stronger Kafka-specific tooling. JSON generally consumes more CPU, bandwidth, and storage. Benchmark representative payloads rather than relying on fixed size ratios.
Connect and Streams: Use compatible Avro, Protobuf, or JSON Schema converters consistently. Kafka Streams state stores and changelog topics also require compatible serialization.
Security: Protect Schema Registry with TLS and authentication, restrict schema-write permissions, and apply Kafka ACLs to topics. Revalidate untrusted JSON because permissive schemas may allow unexpected properties or oversized values.
Monitoring: Track serialization failures, registry latency, compatibility rejections, broker request latency, consumer lag, record size, and transaction abort rates.
Retention and recovery: Configure topic replication and retention independently of the schema format. Schema Registry persists schemas in Kafka’s internal _schemas topic; ensure Kafka backup and disaster-recovery procedures cover that metadata and test restoration.
Operational Best Practices
Register schemas in CI/CD before deploying producers; avoid automatic registration in production.
Exercise old and new schema versions against representative consumers in compatibility tests.
Give every domain a deliberate subject naming strategy and ownership model.
Set explicit compatibility policies and require review for breaking changes.
Cache schemas locally, but design for registry outages, unknown schema IDs, and new schema registration failures.
Keep each transactional.id stable across restarts of the same logical producer while ensuring that concurrent producer instances do not share it.
During incidents, check schema compatibility and serialization errors before assuming the broker or consumer group caused the failure.
Upgrade clients and Schema Registry through rolling tests, retaining previous schema versions until all consumers have migrated.
51
What are schema compatibility modes in Confluent Schema Registry?
KnownRevise
Schema Registry compatibility modes decide whether a new schema version may be registered for a subject. The registry compares the candidate schema against previously registered versions and rejects the registration (HTTP 409) if the change would break consumers or producers that still use the older schema. The available modes are BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE, plus NONE to disable checking.
Read full answerHide full answer
Applies toConfluent Schema Registry 7.x with Apache Kafka 3.x (also the model used by Apicurio Registry and Karapace)
How the Check Works
Schemas are stored per subject (e.g. orders-value), each subject has one schema type (AVRO, JSON, PROTOBUF) and an ordered list of versions.
On POST /subjects/{subject}/versions, Schema Registry runs the compatibility checker for that subject’s type: it resolves the candidate schema against the stored versions according to the subject’s mode.
Non-transitive modes compare only against the latest version; transitive modes compare against every previous version.
The mode is set per subject (PUT /config/{subject}); the registry-wide default comes from schema-registry.properties (schema.compatibility.level; in Docker images it is set with SCHEMA_REGISTRY_SCHEMA_COMPATIBILITY_LEVEL). The default is BACKWARD.
Changing the mode is metadata-only: already-registered versions are not re-validated, so tightening the mode does not retroactively invalidate anything.
The Modes
Mode
Compared against
Meaning
BACKWARD
latest version
New schema can read data written with the previous schema (new consumers can read old data)
BACKWARD_TRANSITIVE
all versions
Same rule, but checked against every historical version
FORWARD
latest version
Previous schema can read data written with the new schema (old consumers keep working with new producers)
FORWARD_TRANSITIVE
all versions
Same rule, checked against every historical version
FULL
latest version
Both backward and forward compatible
FULL_TRANSITIVE
all versions
Both directions, checked against every historical version
NONE
—
No checks; any schema is accepted
Choosing a Mode by Schema Type
Change
BACKWARD
FORWARD
Notes
Avro: add field with default
Yes
Yes
Compatible both ways (FULL): new readers use the default, old readers ignore the field
Avro: add field without default
No
Yes
New readers have no value for old data; old readers just ignore the extra field
Avro: remove field
Yes
Only if the old schema had a default for it
Otherwise the old reader cannot reconstruct the value
Avro: add enum symbol
Yes
Only if the old (reader) schema defines an enum default
Old readers fall back to their default for unknown symbols
Avro: rename field
No
No
Use aliases instead
Protobuf: add optional field
Yes
Yes
Never reuse a deleted field number
JSON Schema: add optional property
Yes
Yes
Adding to required breaks backward compatibility
Configuring and Testing with REST
# Current mode for a subject
curl -s http://schema-registry:8081/config/orders-value
# Set the mode for a subject (or PUT /config for the registry-wide default)
curl -s -X PUT -H 'Content-Type: application/json' \
--data '{"compatibility":"FULL_TRANSITIVE"}' \
http://schema-registry:8081/config/orders-value
# Dry-run a candidate schema: nothing is stored
curl -s -X POST -H 'Content-Type: application/json' \
--data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total\",\"type\":\"double\",\"default\":0.0}]}"}' \
'http://schema-registry:8081/compatibility/subjects/orders-value/versions/latest?testCompatibility=true'
# Register the schema for real
curl -s -X POST -H 'Content-Type: application/json' \
--data '{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"total\",\"type\":\"double\",\"default\":0.0}]}"}' \
http://schema-registry:8081/subjects/orders-value/versions
Java Code Example
import io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient;
import io.confluent.kafka.schemaregistry.client.SchemaRegistryClient;
public class CompatAdmin {
public static void main(String[] args) throws Exception {
SchemaRegistryClient client =
new CachedSchemaRegistryClient("http://schema-registry:8081", 100);
// Tighten the mode before anyone registers the next version
client.updateCompatibility("orders-value", "FULL_TRANSITIVE");
// Producers/consumers can opt out of auto-registration so bad schemas fail fast
System.out.println("Mode set for orders-value");
}
}
Operational Guidance and Pitfalls
Use BACKWARD_TRANSITIVE or FULL_TRANSITIVE for long-lived subjects: it prevents “schema drift”, where each version is only compatible with its immediate predecessor but breaks consumers on version 1. The cost is that the rule tightens as history grows, so plan migrations early.
FULL (non-transitive) is a common middle ground: it protects both directions for the current version while staying easier to evolve than the transitive variant.
Set modes per subject, not globally: unrelated subjects often need different rules (for example, an internal compacted topic may use NONE).
Roll out in order: deploy consumers that can read both versions first, then register the new schema, then switch producers to it.
Set auto.register.schemas=false on serializers in production so an incompatible schema fails at startup instead of being silently registered.
Monitor: alert on registration responses with HTTP 409, and watch the internal _schemas topic for compaction and replication lag. Run Schema Registry in HA (3+ nodes) with kafkastore.bootstrap.servers pointing at a production cluster, and back up the _schemas topic — it is the source of truth for all schema history.
Common traps: Avro fields without defaults, protobuf field-number reuse, JSON Schema required changes, and assuming a mode change re-validates existing versions.
52
How does the Kafka Streams library process event streams?
KnownRevise
Kafka Streams is a Java library for building stateful, real-time applications that consume from and produce to Apache Kafka topics. It runs inside an application’s JVM, with multiple instances coordinating through Kafka.
Read full answerHide full answer
Applies toApache Kafka 4.x (Kafka Streams 4.x)
Architecture and Programming Model
Kafka architecture: Topics are partitioned and replicated across brokers. Kafka 4.x uses KRaft consensus rather than ZooKeeper.
Topology: An application defines a directed graph of source, processor, sink, and join nodes through the high-level DSL or lower-level Processor API.
Streams and tables: A stream represents individual events; a table represents the latest value for each key. The stream-table duality lets the same data be processed using either abstraction.
Partitions and tasks: Partitions are divided into independent stream tasks. Each task processes records sequentially, while multiple task threads provide parallelism.
Consumer groups: Instances sharing an application.id form a consumer group and divide source partitions. Additional instances increase parallelism only when enough partitions exist.
State stores: Joins and aggregations use local state, commonly RocksDB. Kafka changelog topics provide durable recovery data.
Repartitioning: Keys must reach the same partition for key-based joins and aggregations. Kafka Streams creates repartition topics when the topology requires them.
Schema Registry: Kafka Streams does not require Schema Registry. Schema-aware Avro or Protobuf serializers and deserializers can be configured independently, with compatibility enforced by the registry.
State, Ordering, and Delivery Guarantees
Ordering: Kafka preserves order within a partition, not globally across a topic. Ordered processing therefore requires appropriate partitioning.
At-least-once processing: With processing.guarantee=at_least_once, failures can cause records to be processed again. A crash after producing output but before committing the consumed offset can create duplicate output.
Exactly-once processing: In Kafka 4.x, processing.guarantee=exactly_once_v2 is the default. Kafka Streams transactionally commits output records, state-store changelog updates, and consumed offsets.
Transactional requirements: The broker must support transactions. Consumers of transactional source or output topics should use isolation.level=read_committed to filter aborted records.
Scope: Exactly-once applies to Kafka Streams processing, including Kafka-backed state and Kafka output. External side effects require idempotency or external transaction coordination.
Commit interval: commit.interval.ms trades transaction overhead against replayed work after failure. Smaller intervals commit more frequently but increase overhead.
Production Operations and Failure Modes
State recovery: RocksDB files are local working state, not durable backups. After restart, Kafka Streams restores its latest state from changelog topics and replays subsequent records.
Replication: Set the internal-topic replication.factor=3 where appropriate, and verify broker settings such as min.insync.replicas and replication policies.
Scaling: Increase source partitions before adding instances. Too few partitions leave instances idle, while a hot partition cannot be subdivided further.
Rebalancing: Restarts and membership changes trigger consumer-group rebalancing, partition revocation or reassignment among members, and state restoration. Monitor rebalance duration and restoration cost.
Backpressure: Processing lag grows when input exceeds throughput. Tune max.poll.records, task parallelism, and processing performance rather than relying only on lag alerts.
Monitoring: Track source lag, processed-record rate, processing latency, commit failures, rebalances, exceptions, changelog growth, and state-store disk usage through JMX or the metrics platform.
Upgrades: Follow broker and client compatibility requirements. Test changelog restoration, topology changes, and rolling restarts in staging before production.
Backups: Replicated Kafka clusters protect changelog data, but local RocksDB directories are not backups. Back up internal-topic configuration, application configuration, and ACLs.
Poison records: Configure deserialization and processing exception handling, route irrecoverable records to a dead-letter topic, and alert on skipped or failed records.
Kafka Streams Compared with Other Kafka Tools
Tool
Deployment model
Stateful processing
Typical use
Kafka Streams
Embedded in a Java application
Local stores with Kafka changelogs
Business logic, aggregations, joins, transformations
Kafka Connect
Connector workers
Connector-specific
Moving data between Kafka and external systems
Apache Flink
Dedicated distributed runtime
Distributed managed state and checkpoints
Large-scale processing with cluster-level control
Kafka Streams fits when processing logic belongs with a Java service and Kafka is already the system of record. Flink is often preferred for very large state, dedicated cluster scheduling, and independent checkpointing control.
Security and Engineering Trade-offs
Transport security: Use TLS, commonly with security.protocol=SASL_SSL, and configure the broker truststore. Select a supported mechanism such as SCRAM-SHA-512.
Credentials: Supply sasl.jaas.config through secret management rather than committing credentials to configuration files.
Authorization: Grant principals only required topic, group, internal-topic, and transactional-ID permissions.
Transactions: Ensure transaction.timeout.ms does not exceed the broker’s transaction.max.timeout.ms, and monitor transaction timeouts.
Language and operations: Kafka Streams has a native Java API, simplifying deployment but making JVM sizing, thread configuration, and local disk capacity application responsibilities.
Failure isolation: A continuously failing topology can block its partitions. Use supervised deployments, bounded retries, dead-letter handling, and alerts distinguishing broker failures from deterministic data errors.
53
What is the difference between KStream and KTable in Kafka Streams?
KnownRevise
KStream represents an ordered event stream, while KTable represents the latest value for each key. KStream preserves every record independently; KTable materializes state so newer values replace older ones.
Read full answerHide full answer
Applies toApache Kafka 4.x (Kafka Streams 4.x)
Core Distinction
Aspect
KStream
KTable
Data model
Sequence of key-value events
Current key-value state
Duplicate keys
Retained as separate records
Collapsed into the latest value
Historical data
Retained in the source topic according to its retention policy
Not retained in materialized state; the changelog contains update records until compaction removes obsolete versions
Typical use case
Transactions, clicks, measurements, logs
Profiles, account balances, inventory levels, device state
Internal topics
May require repartition topics for stateful aggregations or joins
State-store changelog, normally compacted
Update behavior
Every record is an event
Each update replaces the previous value for that key
For example, sensor readings such as temperature=21, temperature=22, and temperature=23 are three KStream events. The corresponding KTable contains only the latest state: temperature=23.
How They Work Internally
KStream: Each input record is processed according to its partition and offset. A stream does not automatically merge records with the same key; that aggregation must be defined explicitly.
KTable: Kafka Streams maintains a state store indexed by key. A new record updates the stored value. When another instance needs to restore the table, it reads the table’s changelog topic.
Changelog compaction: KTable changelog topics use log compaction to retain the most recent non-null value for each key. Intermediate updates may remain before compaction runs, so storage can temporarily exceed the logical table size.
Tombstones: A null value represents deletion in materialized KTable state and produces a tombstone in the changelog. Compaction does not immediately remove a tombstone; it is retained for the topic’s delete.retention.ms and removed afterward.
KTable is therefore an abstraction over changing state, not simply a compacted Kafka topic. Kafka Streams maintains its changelog for restoration and recovery; downstream applications should normally consume application output topics rather than depend on internal changelog topics.
Aggregation and Joins
After grouping records by key, a KStream can be aggregated into a KTable whose result is backed by a state store.
Joins differ depending on the joined abstraction:
Join
Operational behavior
KStream–KTable
Each stream record is matched against the current table value
KTable–KTable
Emits when either side’s materialized value changes
KStream–KStream
Matches records from the two streams within the join window; repartitioning may be required to co-partition keys
For joins involving stateful KTables, Kafka Streams uses the table’s materialized state and maintains its changelog. Stream–stream joins and certain stream aggregations may introduce repartition topics to ensure records with the same key reach the same task.
Delivery and Ordering Guarantees
Ordering: KStream records are ordered within a partition. A KTable is logically a latest-value view of records processed in that order.
At-least-once: Processing can repeat after failures, so downstream consumers should be idempotent when necessary.
Exactly-once: Kafka Streams supports processing.guarantee=exactly_once_v2 with a unique, stable application.id; Kafka Streams manages transaction IDs internally. Input offsets, state-store changelog writes, and output records are then committed atomically to Kafka, while local state stores are restored consistently after failures.
Limit: Exactly-once does not automatically make an external database or HTTP side effect atomic with Kafka. Such integrations still require idempotency, transactional outbox patterns, or equivalent coordination.
Capacity plan: Changelog topics and restored state require disk capacity. Compaction reduces retained history but does not guarantee immediate disk reclamation.
Recovery: Configure standby replicas for important state stores so another instance can restore them faster and reduce recovery time.
Schema compatibility: Use compatible schemas, typically with Schema Registry, before evolving KStream or KTable payloads.
Choose deliberately: Use KStream for independent facts and event processing; use KTable when consumers care about current state rather than every transition.
54
How do Kafka Streams state stores enable stateful processing?
KnownRevise
State stores let Kafka Streams maintain task-local state while processing records, enabling keyed aggregations, joins, windowed calculations, and deduplication without holding the entire dataset in the application. Persistent, logging-enabled stores normally publish state changes to a Kafka changelog so state can be restored after restarts or task reassignment.
Read full answerHide full answer
Applies toApache Kafka 4.x (Kafka Streams 4.x)
How It Works
A stream task owns one or more input-topic partitions and the corresponding partitions of the state stores it uses:
A record is read from an input topic.
The processor reads or updates state through its store.
The updated state is written to the local store and, for a persistent logging-enabled store, to its changelog topic.
Output records are sent to downstream topics.
Consumed offsets are committed according to the configured processing guarantee.
State is generally partitioned by record key, so all records for a key are handled by the same task and can access the same state partition. Store names must be unique within the topology.
When num.standby.replicas is greater than zero, a standby task maintains a copy of an active task’s state by consuming its changelog. If the active task fails, the standby can be promoted with local state already close to current, reducing restoration work.
Store Types and Changelogs
RocksDB: The default persistent key-value store. It keeps state on local disk and is appropriate when state should not all reside in the Java heap, subject to adequate disk and I/O capacity.
In-memory stores: Faster for small or temporary state. Their contents disappear with the process unless logging and recovery are enabled.
Window stores: Retain timestamped values according to window boundaries and retention settings.
Session stores: Group records into sessions using inactivity-gap and grace-period rules.
A changelog topic is typically named:
<application.id>-<store-name>-changelog
Changelog topics normally use log compaction so that the latest value for each key can be restored. A null-value tombstone removes the key when the changelog record is processed; compaction later removes older non-null records and eventually removes the tombstone itself according to delete-retention settings.
Changelog retention is independent of window-store retention. Operators must retain enough changelog history for the required recovery horizon; if the necessary records expire, Kafka Streams may not be able to rebuild the complete local state.
Recovery Architecture
In Kafka 4.x, broker clusters use KRaft, and changelog durability depends on normal Kafka topic replication. The local RocksDB directory is therefore not a complete disaster-recovery backup. If local state is lost, Kafka Streams can restore it by replaying the changelog.
Production recovery plans should:
Set statestore.changelog.topic.replication.factor appropriately for the broker count.
Monitor replication, under-replicated partitions, disk capacity, and standby-task lag.
Keep state.dir on reliable storage with sufficient capacity and I/O performance.
Protect internal topics with appropriate ACLs.
Use a replicated Kafka cluster, MirrorMaker, or an equivalent backup strategy for cluster-level recovery.
Delivery Semantics and Ordering
Concern
Kafka Streams behavior
Default processing guarantee
At-least-once processing; a crash may cause a record to be processed again
processing.guarantee=exactly_once_v2
Kafka output records, changelog writes and consumed offsets are committed atomically (exactly_once_v2 is the only exactly-once value in Kafka 4.x)
Ordering
Records are processed in order within each input partition
Multiple partitions
No global ordering is provided
Exactly-once semantics apply to Kafka-to-Kafka processing. Local and changelog state are made consistent through Kafka Streams’ checkpoint and restoration machinery; the changelog should not be treated as an exactly-once side effect for an external system. Output consumers should use isolation.level=read_committed to avoid observing aborted transactions.
Exactly-once operation also requires suitable broker transaction-state-log settings, including transaction.state.log.replication.factor and transaction.state.log.min.isr, plus the necessary transactional producer and ACL permissions. Writes to external databases or HTTP systems require separate idempotency or transaction mechanisms.
Production Practices and Failure Modes
Monitor task health: Track input lag, standby lag, processing latency, commit latency, RocksDB metrics, and state-directory disk usage.
Control resource usage: Excessive cache pressure, compaction backlog, or slow disks can cause task lag and restarts.
Plan rebalances: Adding instances or changing stream-thread and partition assignments can trigger state movement and restoration.
Handle tombstones deliberately: Configure serializers and application logic so null values mean deletion rather than accidental data loss.
Secure the platform: Use TLS and SASL, restrict access to application and internal topics, and protect sensitive state at rest.
Test upgrades and recovery: Verify rolling upgrades, restart restoration, partition reassignment, and full state reconstruction rather than assuming local RocksDB files are sufficient.
55
How does Kafka Streams aggregate records within event-time windows?
KnownRevise
Event-time windowed aggregation is Kafka Streams’ mechanism for grouping records by key and timestamp-defined windows while maintaining one aggregate per key-window pair. It uses local state stores for active state and changelog topics to recover or restore that state after failures and rebalances.
Read full answerHide full answer
Applies toApache Kafka Streams 4.x
Core APIs and Window Types
A typical aggregation flow uses groupByKey(), followed by windowedBy(...), and then an aggregation operation such as aggregate(...), reduce(...), or count(). The result is a KTable<Windowed<K>, VR>, where VR is the aggregate’s value type. For example, a five-minute tumbling window can maintain the total order value for each customer in each interval.
Window type
Behaviour
Relevant API
Tumbling
Fixed, non-overlapping intervals aligned to the epoch
TimeWindows.ofSizeWithNoGrace(...) or TimeWindows.ofSizeAndGrace(...)
Hopping
Fixed-size windows that overlap by a configured advance interval
TimeWindows.ofSizeWithNoGrace(...).advanceBy(...) or TimeWindows.ofSizeAndGrace(...).advanceBy(...)
Session
Per-key activity windows separated by an inactivity gap
SessionWindows.withInactivityGapAndGrace(...)
A grace period extends a window after its normal end so that late records can still update the aggregate. Increasing grace improves late-data tolerance, but retains state longer and delays the final result.
Runtime Lifecycle
A stream task consumes a record and uses the input record’s timestamp—not processing time—to assign it to the applicable window or windows.
The task looks up the previous aggregate for that key and window in its local state store.
If no aggregate exists, Kafka Streams invokes the initializer. It then applies the aggregator to the record and the previous aggregate.
The updated aggregate is written to the local store and, for materialized state, recorded in a changelog topic.
The logical KTable exposes the update for downstream processing or materialization.
Kafka Streams derives stream time from observed record timestamps while accounting for out-of-orderness. A record is late and dropped when stream time has already passed the end of its window plus the grace period. Session-window grace begins after the per-key inactivity gap ends.
Without suppression, downstream consumers can receive several updates for one window. Suppressed.untilWindowCloses(...) can defer time-windowed results until the window closes, reducing intermediate output.
Delivery Semantics and Guarantees
Default processing: Kafka Streams normally provides at-least-once processing. After a failure, a record may be processed again, so aggregators and side effects should be idempotent where possible.
Exactly-once processing: Set processing.guarantee=exactly_once_v2 to atomically coordinate consumed offsets, Streams state changes, and Kafka output records. Use an application.id that uniquely identifies the application’s state and a stable state.dir. The broker should also use appropriate transaction.state.log.replication.factor and transaction.state.log.min.isr settings for transaction-state durability.
External side effects: Exactly-once processing does not automatically make non-transactional effects, such as an HTTP request or SQL update, exactly once.
Ordering: Kafka preserves record order within a partition. Records for one key must therefore reach the same partition when their order matters.
Operational Behaviour
Hot keys: A highly active key can skew one partition and one state-store partition, limiting scalability.
State growth: Long windows, large grace periods, and high-cardinality keys increase RocksDB disk use, cache pressure, and restore time.
Recovery: After a failure or rebalance, restoring a large changelog can delay processing while the affected task becomes active.
Retention: Changelog retention must protect the state needed for recovery. Input retention is a separate concern for replaying source data, and changelog compaction does not replace correct retention and state-lifecycle planning.
Durability: Use replicated topics, an appropriate min.insync.replicas, and producer durability settings matched to the required loss tolerance.
Production Guidance
Monitor consumer lag, processing latency, rebalance duration, state-store size, local disk pressure, and restore time.
Size grace periods from measured lateness rather than choosing an unnecessarily large default.
Test failure recovery, rebalances, broker replacement, and rolling upgrades with production-like topic and key distributions.
Verify exactly-once behaviour end to end when it is required; transaction guarantees apply only to Kafka-backed inputs, outputs, and Streams state.
56
How does ksqlDB process streaming data?
KnownRevise
ksqlDB is a streaming database that executes SQL continuously over Apache Kafka records. It transforms, joins, filters, and aggregates events, maintains state where necessary, and publishes results back to Kafka.
Read full answerHide full answer
Applies toApache Kafka 3.9.x with ksqlDB 0.29.x
Architecture and Data Model
Kafka brokers store partitioned, replicated topic logs; in KRaft mode, a controller quorum manages cluster metadata. ksqlDB remains a query service and Kafka client, not a replacement for those brokers or controllers.
Stream: An append-only sequence of events; each row is an independent fact, and a key is optional.
Table: A mutable view holding the latest value per key, built from a changelog or compacted topic or by an aggregation.
Persistent query: CREATE STREAM ... AS SELECT or CREATE TABLE ... AS SELECT. It runs continuously on the server and writes its results to a Kafka topic.
Push query: SELECT ... EMIT CHANGES. It streams new results to the client as events arrive until the client stops it; nothing is written to Kafka.
Pull query: SELECT without EMIT CHANGES. It returns the current state once and ends, like a database query, usually as a key lookup against a materialized table.
ksqlDB uses the Kafka Streams runtime internally. Each persistent query is a Kafka Streams application with its own application ID and consumer coordination; source partitions and offsets are managed through Kafka. Stateful queries keep local state, normally in a RocksDB-backed state store, and use compacted changelog topics to restore state after failures.
Push and Pull Queries
Property
Pull query
Push query
Execution
Runs once and terminates
Runs until the client cancels it
Output
Current result returned to the client
A continuous stream of changes sent to the client
State
Reads the state store of a table materialized by a persistent query (or scans a stream’s topic)
Subscribes to new events; the state itself is maintained by persistent queries
Typical use
Point lookups, such as a customer’s current balance
Live dashboards and notifications
Consistency
Current per-key result; no global cross-partition snapshot
Incremental updates as new records arrive
Pull queries are cheapest as key lookups on a materialized table. A pull query that scans a whole table or stream reads every partition and is expensive. Neither form provides a globally consistent snapshot across partitions.
Delivery, Ordering, and Exactly-Once Semantics
Kafka preserves record order only within a partition. ksqlDB preserves that order until an operation such as a repartitioning join changes the partitioning.
Grouped aggregations emit updates by grouping key; there is no global ordering across output partitions.
By default ksqlDB queries are at-least-once, so a failure can publish duplicate output records.
For exactly-once processing inside Kafka, set ksql.streams.processing.guarantee=exactly_once_v2 in the server configuration; ksqlDB then uses Kafka Streams transactions for its output, state and consumed offsets. The brokers need a replicated transaction-state log.
Downstream consumers should use isolation.level=read_committed so they do not read aborted output. Data that Kafka Connect brings in or writes out is exactly-once only if those connectors support it.
Exactly-once applies to the transactional processing chain, not automatically to arbitrary external side effects performed by a connector or application.
Kafka Connect and Schema Registry
Kafka Connect supplies source and sink connectors for databases, logs, files, SaaS systems, and other external systems. ksqlDB can use an embedded Connect deployment or a separately operated distributed cluster. Connector lag, task failures, retries, and transactional support must be monitored.
Schema Registry stores and validates schemas used by JSON_SR and AVRO_SR. The configured compatibility policy, such as BACKWARD, BACKWARD_TRANSITIVE, or FULL, controls which schema changes are accepted. Unregistered or incompatible records can cause deserialization failures.
Production Operations, Failures, and Security
Monitoring: Track query status, processing lag, state-store size, changelog lag, broker disk and request latency, Connect task lag, and schema errors.
Capacity: Increase partitions for independent producer throughput, and size state stores, changelog retention, heap, and RocksDB resources for the working set.
Retention and compaction: Time or size retention bounds log growth; compaction retains the latest value per key for compacted topics. Compaction is not a backup mechanism, and retention still limits historical replay.
Recovery: Protect Kafka and Schema Registry with appropriate replication and backups. Push-query state can normally be rebuilt by restoring the source and changelog data, subject to retention.
Upgrades: Test schema, connector, and query compatibility, then upgrade ksqlDB instances in a controlled rolling sequence while verifying state-store recovery.
Security: Use TLS or mTLS, SASL authentication, Kafka ACLs, authenticated Schema Registry access, and encrypted ksqlDB listeners. Avoid placing credentials directly in shell history or process arguments.
Failure modes: Do not assume global ordering, globally consistent pull-query snapshots, or connector-wide exactly-once delivery unless the required transactional configuration is present.
57
Kafka Streams vs. Apache Flink and Spark Streaming
KnownRevise
Kafka Streams is an embedded stream-processing library for Java/Kotlin services, whereas Apache Flink is a distributed stream-processing platform and Spark Structured Streaming is Spark’s incremental streaming API. The main trade-off is application-level simplicity versus cluster-level scale and operational control.
Read full answerHide full answer
Applies toApache Kafka 4.x with Kafka Streams, Apache Flink 2.x, and Apache Spark 4.x with Structured Streaming
Architecture and Guarantees
Dimension
Kafka Streams
Apache Flink
Spark Structured Streaming
Deployment
Embedded in an application
Distributed cluster with JobManager and TaskManagers
Spark cluster or managed platform
Execution
One task per input partition, run by a configurable number of stream threads; record-at-a-time
Tasks and subtasks scheduled across workers; compatible operators may be chained
Micro-batches by default; continuous processing mode is available for supported sources
Ordering
Preserves input-record order within a partition
Preserves keyed order within appropriately partitioned keyed streams
Depends on source and partitioning; no global ordering
State and recovery
Usually RocksDB local stores with Kafka changelog restoration
Pluggable state backends with distributed checkpoints and savepoints
HDFS-backed (default) or RocksDB state store plus checkpointed metadata and source replay
APIs
Java, Kotlin and Scala
Java, Python (PyFlink), and SQL; the Scala API was removed in Flink 2.0
Java, Scala, Python, and SQL
Kafka Streams: Each input partition becomes a task, and tasks are spread over the instance’s stream threads. Stateful operators maintain local state, and state changes are written to changelog topics so another instance can restore them.
Flink: A job graph is divided into tasks and subtasks. Checkpoints provide consistent distributed snapshots, while savepoints support planned upgrades and state migration.
Spark Structured Streaming: A streaming query is represented as an unbounded table or streaming DataFrame. Checkpoints store offsets and query progress, but recovery also depends on the source supporting replay.
Kafka Streams delivery semantics: processing.guarantee accepts at_least_once or exactly_once_v2 (the older exactly_once values were removed in Kafka 4.0). In exactly_once_v2 mode, consumed offsets and output records are committed atomically through Kafka transactions. Kafka consumers that must not observe aborted records should use isolation.level=read_committed.
Flink delivery semantics: End-to-end exactly-once generally requires checkpointing, deterministic processing, replayable sources, and a transactional or idempotent sink.
Spark delivery semantics: Exactly-once processing requires checkpointing, a replayable source, and a sink that supports the required delivery contract; otherwise failures can produce duplicates or lost results.
Exactly-once processing does not automatically make external side effects exactly once. Database writes, HTTP requests, and non-transactional destinations can be repeated and need idempotency, transactional participation, deduplication, or an outbox pattern.
State and Workload Fit
Kafka Streams: Best for low-latency transformations, filtering, routing, aggregations, joins, and Kafka-to-Kafka processing within an existing service. It requires little additional infrastructure, although stateful applications need careful key design, changelog retention, and capacity planning.
Flink: Strong fit for complex event-time windows, timers, joins, asynchronous I/O, very large distributed state, and workloads requiring explicit checkpoint and rescaling control.
Spark Structured Streaming: Strong fit for Spark SQL, DataFrame transformations, analytics, and machine-learning workflows that reuse existing batch code and tolerate the latency of the selected execution mode.
Interactive processing: Kafka Streams exposes Interactive Queries over its local state, which you wrap in your own HTTP service. Flink and Spark have stronger native SQL integration and broader data-platform tooling.
Production Operations
Monitoring: Use Kafka Streams JMX metrics, consumer-lag metrics, and application logs; Flink Dashboard, Prometheus metrics, and logs; and Spark UI, metrics, and structured logs. Track lag, state size, skew, checkpoint duration, backpressure, and resource pressure.
Hot partitions: A popular key can overload one partition. Kafka Streams cannot split one key’s state arbitrarily without changing the partitioning model; use explicit repartitioning or redesign the key distribution.
Rebalances: Kafka Streams group changes can temporarily pause processing. Cooperative rebalancing reduces disruption but does not eliminate assignment latency.
State growth: Unbounded keys and long-lived windows can exhaust disk or memory. Monitor state stores, changelog lag, checkpoint sizes, and compaction behavior.
Failure recovery: Keep enough capacity for Kafka Streams standby instances, test changelog restoration, and regularly test Flink savepoint recovery and Spark checkpoint recovery. Kafka Streams upgrades normally use rolling deployments; Flink commonly uses savepoints; Spark requires checkpoint compatibility testing.
Security: Use TLS, authentication, authorization, encrypted checkpoint or savepoint storage, secret management, and restricted administrative interfaces. SCRAM or Kerberos is commonly used for Kafka authentication.
When to Choose Each
Choose Kafka Streams for application-level Kafka processing where low latency and avoiding another cluster matter most.
Choose Flink for event-time-heavy computation, large distributed state, advanced joins, and explicit checkpoint and rescaling control.
Choose Spark Structured Streaming for SQL, analytics, machine learning, and reuse of Spark DataFrame or batch-processing code.
A practical rule is to prefer Kafka Streams for embedded service processing, Flink for latency-sensitive distributed event-time processing, and Spark Structured Streaming for unified analytics and machine-learning pipelines.
58
How do you secure Apache Kafka with TLS, SASL, and ACLs?
KnownRevise
Secure Apache Kafka by combining TLS for transport encryption, SASL for authentication, and ACLs for topic-level authorization. These controls should be applied to client listeners, inter-broker traffic, and KRaft controller traffic.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode
Security Model
TLS protects confidentiality and integrity, but does not normally identify clients by itself.
SASL authenticates clients and brokers. Prefer SCRAM-SHA-512; use PLAIN only with TLS and where stronger mechanisms are unavailable.
ACLs authorize authenticated principals to perform operations on topics, consumer groups, transactional IDs, and cluster resources.
Use SASL_SSL rather than SASL_PLAINTEXT. Optional mutual TLS adds certificate-based client authentication, but it does not replace ACLs.
Server Configuration
A minimal combined broker/controller configuration looks like this:
The broker certificate must contain kafka1.example.com in its SAN. The controller listener uses mutual TLS here, so the certificate’s principal (User:CN=kafka1.example.com) is listed in super.users along with the inter-broker SCRAM user. On brokers, the JAAS setting must carry the listener and mechanism prefix, as shown. In production, use separate controller and broker nodes, a three- or five-controller quorum, and inject passwords from a secret manager rather than storing them directly in server.properties.
SASL Users and ACLs
SCRAM credentials are stored as salted, iterated hashes in Kafka metadata. Create the inter-broker and application users with kafka-configs.sh:
Since Kafka 3.0, idempotent producers need only Write on the topic; transactional producers also need Write and Describe on their transactional.id. Use a distinct service account for each workload, and never make application accounts superusers.
Code Example
A confluent-kafka Python client uses the same transport controls:
from confluent_kafka import Producer
errors = []
def delivered(err, _message):
if err is not None:
errors.append(str(err))
producer = Producer({
"bootstrap.servers": "kafka1.example.com:9093",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "SCRAM-SHA-512",
"sasl.username": "app-writer",
"sasl.password": "CHANGE_ME",
"ssl.ca.location": "/etc/kafka/ca.crt",
"enable.idempotence": True,
"acks": "all",
})
producer.produce(
"payments",
key=b"order-42",
value=b"created",
on_delivery=delivered,
)
remaining = producer.flush(10)
if remaining:
raise TimeoutError("Kafka delivery did not complete")
if errors:
raise RuntimeError(errors[0])
Failure Modes and Operations
TLS handshake failure: check certificate expiry, CA trust, SAN/hostname mismatch, protocol negotiation, and advertised.listeners.
SASL authentication failure: verify that client and broker mechanisms match and that the SCRAM username exists. Clients may receive only a generic authentication error, so inspect broker logs.
Authorization failure: inspect ACLs with kafka-acls.sh --describe; DENY ACLs take precedence over ALLOW ACLs.
Rotation: distribute a new CA before rotating leaf certificates, then perform a rolling restart. Rotate SCRAM credentials carefully because clients and servers must remain synchronized.
Monitoring: alert on handshake failures, authentication failures, authorization denials, certificate expiry, under-replicated partitions, and consumer lag.
Disaster recovery: back up server.properties, JAAS configuration, certificates, and secret references. ACL and SCRAM user metadata is replicated as KRaft metadata, not stored in ordinary topic data.
Bootstrap requires special care: the inter-broker SCRAM user must exist before brokers can authenticate to each other. In KRaft, create it when formatting storage, for example kafka-storage.sh format --cluster-id <id> --config server.properties --add-scram 'SCRAM-SHA-512=[name=broker,password=CHANGE_ME]' (Kafka 3.5+), then create the remaining users with kafka-configs.sh. No production listener should remain on PLAINTEXT.
59
How do you monitor an Apache Kafka cluster?
KnownRevise
Monitor Apache Kafka with layered observability: broker and controller metrics, client telemetry, consumer-lag checks, logs, and infrastructure metrics. In KRaft mode, verify both data-plane health and controller-quorum health.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode, Prometheus 3.x, and JMX Exporter 1.x
Monitoring Model
Kafka exposes operational data through JMX. A typical production setup exports broker metrics with JMX Exporter, scrapes them with Prometheus, and displays them in Grafana or another observability platform. Client applications should export producer and consumer metrics separately because broker metrics cannot describe every application-level failure.
Monitor these layers:
Data plane: partitions, leaders, in-sync replicas, request queues, disk throughput, and network utilization.
Control plane: KRaft controllers, quorum health, metadata changes, and controller elections.
Clients: producer errors and retries, consumer rebalances, processed-record rates, and committed offsets.
Infrastructure: CPU, memory, garbage collection, disk capacity, I/O latency, and network errors.
Business signals: end-to-end message age, missing expected partitions, and downstream processing delays.
Replication is behind or a partition has no available leader.
Partition state
LeaderCount, PartitionCount, leader-election rate
Detects abnormal imbalance, elections, or topology changes.
Broker health
ActiveControllerCount, RequestHandlerAvgIdlePercent, request queue and latency metrics
Identifies an overloaded broker or unhealthy controller.
Producer
Error rate, retry rate, record queue time, request latency
Detects rejected writes, unavailable brokers, or slow acknowledgements.
Consumer
Consumed-record rate, fetch latency, rebalance count, poll idle time
Detects stalled consumers and unstable group membership.
Consumer lag
Committed offset versus log-end offset
Estimates unprocessed records; also track lag age because record count alone can be misleading.
Infrastructure
Disk free space and inodes, I/O latency, CPU, GC pauses, network errors
Finds Kafka failures caused by resource exhaustion.
Alert on OfflinePartitionsCount > 0, sustained under-replicated partitions, an unavailable controller, broker loss, exhausted disks, excessive request queues, and consumer lag exceeding business-defined time limits. Prefer SLO-based thresholds over fixed record counts because traffic and partition sizes vary.
CLI Checks
Inspect partition assignments, leaders, replicas, and ISR state:
A healthy quorum should report the expected voters and one active controller. CLI commands are useful for diagnosis, but continuous alerting should come from metrics rather than polling these commands.
Code Example
Install JMX Exporter and expose broker metrics on port 7071. Add this to the broker service environment before starting Kafka:
Keep port 7071 on a private management network or place it behind an authenticated TLS reverse proxy. Do not expose raw JMX or an unsecured exporter endpoint publicly.
Delivery Guarantees and Failure Modes
Ordering exists only within a partition, not across an entire topic.
At-least-once processing requires committing a consumer offset after successful processing. A crash between processing and commit can replay records.
At-most-once processing commits before processing and can lose work after a crash.
Exactly-once Kafka processing requires producer transactions, a stable transactional.id, and isolation.level=read_committed. Applications must include the consumed offsets in the same transaction; Kafka cannot make arbitrary external side effects exactly-once.
For durable writes, producers generally use acks=all, idempotence, and a replicated topic with an appropriate min.insync.replicas.
Typical incidents include disk exhaustion preventing replicas from catching up, an ISR shrinking after a network partition, repeated leader elections, controller-quorum instability, consumer rebalance storms, and long producer retries masking broker unavailability.
Production Practices
Monitor per-broker and per-topic cardinality carefully; very large topic counts can increase Prometheus cost.
Keep dashboards for cluster capacity, replication, request latency, producer health, consumer lag, and KRaft health.
Correlate metrics with broker logs and client traces during incidents.
Test alerts by stopping a broker, filling a disk, introducing consumer delay, and simulating controller loss in staging.
Upgrade brokers in a controlled rolling sequence and verify replication, quorum status, and consumer groups after each step.
Use TLS and SASL for client, administrative, and monitoring access; store credentials in a secrets manager and apply least privilege.
60
Which Apache Kafka metrics matter most in production?
KnownRevise
The metrics that matter most in production are broker health, replication status, producer and consumer behaviour, consumer lag, storage pressure, and component error rates. The first alerts should fire on offline or under-replicated partitions, growing consumer lag, rising request latency, and sustained error or retry rates.
Read full answerHide full answer
Applies toApache Kafka 4.x (KRaft mode)
Essential Broker and Replication Metrics
Metric
What it indicates
Why it matters
Under-replicated partitions
In-sync replicas below the configured replication factor
Reduced fault tolerance; possible data loss if another replica fails
Offline partitions
Partitions with no elected leader
Producers and consumers cannot use them
Leader election rate
How often leadership changes
Frequent elections signal broker instability or network disruption
ISR shrink/expand rate
Changes in the in-sync replica set
Repeated shrinking means replicas cannot keep up
Controller status
Whether a KRaft controller quorum is active
Losing quorum blocks metadata changes and recovery
Request latency
Time brokers spend on requests
Rising percentiles point to overload, slow disks or GC pauses
Broker CPU, memory, GC, disk
Resource pressure on brokers
Saturation raises latency and triggers leader movement
Network throughput and errors
Inter-broker and client traffic
Reveals saturation and retransmissions
A healthy cluster has zero offline partitions and no sustained under-replicated partitions; brief increases during rebalancing or replica replacement are normal, while a growing count is an incident.
Retry rate: sustained retries point to unavailable brokers or metadata problems.
Record queue time: how long records wait before transmission; high values mean a CPU-bound producer or slow metadata lookup.
Batch size and compression ratio: tiny batches hurt throughput, oversized batches raise latency and can trigger MessageSizeTooLargeException.
Produce request latency: alert against SLOs, not a fixed threshold, because hardware and message sizes vary.
Idempotence and transaction errors: applications needing duplicate prevention must watch sequence-number errors and aborted transactions.
With acks=all, replication completes only when the required in-sync replicas acknowledge the record, so producer success alone can hide a dangerously small ISR.
Consumer and Consumer-Group Metrics
Consumer lag is the main capacity and freshness metric, computed as log end offset - committed consumer offset. Always break it down per partition: a large aggregate can hide one hot partition.
Poll rate: near zero can mean a stalled consumer.
Records consumed per second: confirms processing progress.
Rebalance count and duration: frequent rebalances cause stop-the-world pauses and duplicate work.
Commit latency and failures: slow or failed commits cause repeated processing and stall lag reduction.
Fetch latency: separates broker-side slowness from application-side delay.
Processing latency: time inside business logic, often the dominant part of end-to-end lag.
Dead-letter queue rate: a growing DLQ means poison messages are being rejected.
A consumer can be alive with healthy heartbeats while lag climbs, so combine liveness, throughput, lag and processing latency.
Storage, Retention and Compaction
Track disk usage per broker and partition, plus:
Log start offset: earliest retained offset.
Log end offset: newest appended offset.
Segment count and size: influences file handles, recovery time and compaction work.
Retention age: how long records stay available.
Compaction lag and tombstone count: matters for compacted topics keyed by current state.
Disk I/O latency and filesystem errors: often the earliest sign of storage trouble.
Sizing must include replicas, segment growth and bursts; hitting the filesystem limit can take a broker down and trigger leader elections.
A Connect sink task can look healthy while its destination is unavailable, so watch sink error and retry rates together. In Kafka Streams, large state stores or slow commits can cap throughput even when broker metrics look fine.
Operational Checks and Alerting
Collect broker, client, Connect, Streams and Schema Registry metrics through JMX or a supported exporter, and correlate them with CPU, disk latency, filesystem capacity, network drops and TLS/authentication failures. Two CLI checks are kafka-topics.sh --bootstrap-server broker1.example.com:9093 --describe --topic orders and kafka-consumer-groups.sh --bootstrap-server broker1.example.com:9093 --describe --group order-processors.
Alert on symptoms and trends, not isolated values: offline partitions, sustained ISR shrinkage, controller quorum loss, rapidly increasing lag, sustained produce/fetch errors, disk exhaustion, and repeated Connect or Streams task failures. Keep broker, client and consumer-owned metrics on separate dashboards so an application bottleneck is not blamed on Kafka.
61
How do you tune Apache Kafka for high throughput?
KnownRevise
Tune Apache Kafka throughput by aligning partition parallelism, producer batching, consumer fetching, replication, and hardware with the workload. More partitions and larger batches can improve throughput, but hot keys, slow disks, excessive partitions, or weakened durability can erase those gains.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode with confluent-kafka 2.x
Primary Throughput Levers
Area
Important settings
Operational trade-off
Partitioning
Topic partitions, consumer-group membership
Partitions provide producer and consumer parallelism. Size them for target throughput and required consumers, not arbitrary scale. A hot key still overloads one partition.
Producers
batch.size, linger.ms, compression.type
Larger batches reduce request overhead; linger.ms waits briefly for more records. Compression reduces network and disk I/O but consumes CPU.
Tune only after identifying CPU, request-queue, replication, or disk bottlenecks. Spread log.dirs across capable disks.
Durability
acks=all, min.insync.replicas, idempotence
Higher durability and replication consume additional network and disk bandwidth. acks=all does not itself provide exactly-once processing.
Kafka’s write path batches records by partition, compresses each batch, sends it to the partition leader, appends it locally, replicates it to followers, and acknowledges it according to the producer’s durability settings. Consumers fetch batches from assigned partitions and coordinate their progress through consumer-group offsets.
Delivery Guarantees and Ordering
Ordering: Kafka preserves record order within a partition, not across an entire topic. Use the partition key as the business ordering key.
Idempotent production: Set enable.idempotence=true to prevent duplicate records caused by producer retries. It does not prevent a consumer from processing a record again after a crash.
At-least-once consumption: Set enable.auto.commit=false, process the records, and then commit their offsets. A crash before the commit causes replay and possible duplicate side effects.
Exactly-once consumption: For consume-transform-produce within Kafka, use a Kafka transaction that includes output records and consumed offsets. Consumers reading transactional output must use isolation.level=read_committed. Kafka Streams processing.guarantee=exactly_once_v2 provides equivalent transaction semantics for Kafka records, state stores, and output. External database writes still require an atomic pattern such as an outbox.
Safe in-flight requests: With idempotence enabled, max.in.flight.requests.per.connection=5 preserves per-partition order while allowing parallel requests.
Production Operations and Failure Modes
Monitor produce and fetch latency, throughput, request queue time, GC pauses, disk latency and utilization, network saturation, under-replicated partitions, offline partitions, and frequent ISR changes.
Use replication factor 3 and min.insync.replicas=2 for important topics. A broker failure then retains write capacity, but losing additional replicas can stop writes.
Rebalances, oversized consumer batches, and slow processing can reduce effective throughput. Keep max.poll.interval.ms compatible with processing time and avoid committing offsets before processing succeeds.
Kafka Connect worker concurrency is controlled by tasks.max; normally use no more tasks than the connector’s partition count. Kafka Streams performance also depends on repartition topics, state-store throughput, and num.stream.threads.
The separate Confluent Schema Registry can centralize Avro and Protobuf schemas. Binary serialization can reduce payload size, and configured compatibility checks can block incompatible changes.
Use TLS with SASL_SSL or mTLS, least-privilege ACLs, and client quotas to limit noisy-neighbor impact.
Replication is not a backup. MirrorMaker 2 can provide cross-cluster replication and disaster recovery, but it is not a substitute for backups; use a separately tested backup and restore mechanism.
Upgrade brokers in a controlled rolling sequence while maintaining the KRaft controller quorum and adequate replica placement.
Code Example
Create a replicated topic; the partition count is an illustrative starting point that must be benchmarked:
Produce records with batching, compression, durable acknowledgements, and idempotence:
python -m pip install 'confluent-kafka~=2.0'
from confluent_kafka import Producer
errors = []
def delivery_report(error, message):
if error is not None:
errors.append(str(error))
producer = Producer({
"bootstrap.servers": "kafka-1.example.com:9093",
"acks": "all",
"enable.idempotence": True,
"compression.type": "lz4",
"batch.size": 131072,
"linger.ms": 20,
"max.in.flight.requests.per.connection": 5,
"delivery.timeout.ms": 120000,
"on_delivery": delivery_report,
})
payload = b"x" * 1024
for sequence in range(10_000):
producer.produce(
"orders",
key=f"order-{sequence}",
value=payload,
)
# flush() returns the number of messages still awaiting delivery.
remaining = producer.flush(30)
if remaining:
raise RuntimeError(f"{remaining} Kafka message(s) were not delivered within 30 seconds")
if errors:
raise RuntimeError("Kafka delivery failed: " + "; ".join(errors))
Tuning Workflow and Pitfalls
Establish baseline throughput, p95/p99 latency, error rate, disk latency, and ISR stability.
Benchmark in staging with representative record sizes, keys, compression, and consumer concurrency.
Increase partitions and brokers only until useful parallelism is reached; changing one variable at a time makes regressions identifiable.
Tune batching and prefetch sizes only after fixing skewed keys, undersized brokers, or slow disks.
Avoid extremely large batches, frequent producer flushes, acks=0 for important records, manual offset commits before processing, and unbounded partition growth.
62
How can Apache Kafka be optimized for low-latency messaging?
KnownRevise
Low-latency Kafka tuning minimizes the delay across producer batching, broker replication, consumer fetching, and application processing while preserving explicit durability and ordering requirements. The largest gains usually come from reducing unnecessary waits, preventing broker and consumer saturation, and measuring p95/p99 latency rather than optimizing averages.
Application processing before the offset is committed.
Ordering is guaranteed only within a partition. Increase partition count to parallelize independent traffic, but excessive partitions increase controller, metadata, replication, and consumer-assignment overhead. Use keys when ordering related messages.
Broker and Storage Tuning
Tune from measurements rather than applying universal values:
Setting or resource
Low-latency approach
Trade-off
num.network.threads
Increase if NetworkProcessorAvgIdlePercent is low
More CPU usage and context switching
num.io.threads
Increase for concurrent replication and disk I/O
Benchmark against disk saturation
num.replica.fetchers
Consider 2 or more on fast disks
More concurrent disk reads
socket.send.buffer.bytes
Test larger values such as 131072
Linux autotuning may already perform better
socket.receive.buffer.bytes
Test larger values such as 131072
More memory per connection
fetch.min.bytes
Use 1 for minimum fetch delay
More frequent, smaller fetch requests
fetch.max.wait.ms
Reduce from 500 to roughly 10-50
Greater broker and network load
Use local NVMe storage, keep Kafka data volumes separate from unrelated workloads, and preserve enough RAM for the OS page cache. Kafka normally relies on the page cache rather than forcing an fsync for every produce request.
Run the broker with Java 17 or newer. Keep the heap below roughly half the container’s memory so page cache remains available; a common starting point is a fixed 6–8 GiB heap for a moderately sized broker.
Producer and Consumer Configuration
Kafka 4.1 Java producer defaults include acks=all, idempotence enabled, and linger.ms=5. For latency-sensitive, high-throughput traffic:
Use batch.size around 32–64 KiB and linger.ms around 1–5 ms to batch requests without excessive waiting.
Use compression.type=lz4 when payloads are large enough for compression to reduce network and broker work.
Set buffer.memory according to the allowed batching concurrency; around 32–64 MiB is a reasonable starting point, not a universal limit.
Keep idempotence enabled so producer retries do not create duplicate records. This does not make downstream consumer processing exactly once.
acks=1 can reduce latency, but a leader failure can lose acknowledged records. Prefer acks=all with at least three replicas and min.insync.replicas=2 for production durability.
For consumers, use fetch.min.bytes=1 and a short fetch.max.wait.ms when minimum latency matters. Limit max.poll.records to prevent large processing batches, while keeping max.poll.interval.ms greater than the worst-case processing time. Use isolation.level=read_committed for transactional records; read_uncommitted is faster but can expose aborted data.
Consumer-side processing is normally at least once: commit offsets only after successful processing, and design handlers to tolerate duplicates. Exactly-once processing requires Kafka transactions, sendOffsetsToTransaction, and read_committed; it does not automatically cover non-Kafka side effects.
Connect, Streams, and Schema Registry
Kafka Connect: tune tasks.max, consumer.max.poll.records, producer.batch.size, and producer.linger.ms. Increasing offset.flush.interval.ms reduces commit overhead but affects recovery-point latency.
Kafka Streams: tune num.stream.threads for parallelism and commit.interval.ms against checkpoint overhead. Lower statestore.cache.max.bytes (even to 0) and commit.interval.ms when each update must be forwarded immediately; a larger cache reduces downstream churn but adds latency.
Schema Registry: use Avro or Protobuf rather than JSON where appropriate. Keep the client’s schema cache enabled to avoid fetching schemas on every request.
Code Example
Create and inspect a low-latency topic on three brokers:
import java.util.Properties;
import java.util.concurrent.TimeUnit;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.serialization.StringSerializer;
public class Produce {
public static void main(String[] args) throws Exception {
Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.example.com:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
props.put(ProducerConfig.COMPRESSION_TYPE_CONFIG, "lz4");
props.put(ProducerConfig.BATCH_SIZE_CONFIG, 65536);
props.put(ProducerConfig.BUFFER_MEMORY_CONFIG, 67108864L);
props.put(ProducerConfig.LINGER_MS_CONFIG, 2);
try (KafkaProducer<String, String> producer = new KafkaProducer<>(props)) {
producer.send(new ProducerRecord<>("orders", "key-42", "payload"))
.get(5, TimeUnit.SECONDS); // Synchronous for a latency benchmark
}
}
}
Production Operations
Monitor p95/p99 end-to-end latency, producer request latency, record-queue-time-avg, consumer fetch latency, consumer lag, NetworkProcessorAvgIdlePercent, RequestHandlerAvgIdlePercent, ISR changes, offline partitions, leader elections, GC pauses, disk latency, and CPU saturation.
Load-test with representative payload sizes, compression, TLS, replicas, and consumer concurrency. A loopback test is not representative.
Investigate long-tail latency through partition skew, hot keys, leader rebalancing, under-replicated partitions, page-cache pressure, and consumer rebalances.
Roll broker changes gradually, verify ISR after each restart, and never remove replicas before replacements have caught up.
Enable TLS and SASL where traffic crosses trust boundaries, restrict topics with ACLs, and include authentication overhead in latency testing.
Replication is not a backup. Mirror critical topics to another cluster or independent storage.
63
Why is Apache Kafka fast?
KnownRevise
Kafka is fast because it combines sequential I/O, the operating system’s page cache, batching, compression, and selective zero-copy transfers. Its append-only partition logs let producers and consumers move data efficiently without database-style record updates or a dense index for every event.
Read full answerHide full answer
Applies toApache Kafka 4.x
Architecture and I/O Path
Topics and partitions: A topic is divided into partitions. Each partition has one leader and optional follower replicas, allowing independent partitions to process traffic in parallel.
Append-only logs: Producers append batches to the leader partition. Consumers track positions with offsets instead of searching for or modifying individual records, keeping the hot read path inexpensive.
Replication: Followers replicate leader data. With acks=all, the leader waits for all current in-sync replicas to acknowledge the write, subject to min.insync.replicas.
KRaft: Kafka 4.x uses KRaft for controller quorum and metadata management; the ZooKeeper-based architecture is no longer supported.
Consumer groups: Group members receive disjoint partitions. Offsets can be committed automatically or, preferably for at-least-once processing, after successful business processing.
Why Kafka Is Fast
Sequential I/O: Each partition is an append-only log, so brokers perform sequential writes rather than random updates. Multiple partitions provide parallel I/O paths.
Page cache: The broker relies primarily on the OS page cache rather than keeping every record in the JVM heap. Data is handed to cached pages and flushed by the operating system; recently written data can also be served efficiently from cache.
Zero-copy: For plaintext transfers, Kafka can use kernel mechanisms such as sendfile, avoiding unnecessary user-space and kernel copies. This is not literally zero copying, and TLS adds CPU and copying costs.
Batching and compression: Producers group records and can compress them with lz4 or zstd. Fewer, larger requests reduce network overhead and improve disk efficiency.
Limited indexing: Offsets identify log positions directly. Kafka maintains sparse indexes where needed rather than a database-style index for every record.
Delivery Guarantees and Trade-offs
Model
Configuration and behavior
At-most-once
The consumer commits before processing; a crash can lose work.
At-least-once
The consumer processes first and commits afterward; a crash can duplicate work.
Exactly-once within Kafka
Requires a transactional producer with transactional.id, transactional offset submission, and consumers using isolation.level=read_committed. External side effects still require idempotency or distributed transaction coordination.
Ordering is guaranteed only within a partition, not across an entire topic. Idempotent producers ensure that retries of the same produce request do not create duplicate records, but idempotence alone does not provide end-to-end exactly-once processing.
Supporting Components
Kafka Connect: Source connectors read external systems, sink connectors write to them, and converters serialize or deserialize records. Record transformations are performed by Single Message Transforms (SMTs), while slow destinations or converters can become bottlenecks.
Kafka Streams: Processes streams and maintains local state, commonly using changelog topics for recovery and repartition topics when repartitioning is required. processing.guarantee=exactly_once_v2 enables Kafka-scoped exactly-once processing.
Schema Registry: Stores schemas and can reject schema-invalid records or enforce compatibility policies, while adding serialization and lookup overhead.
Retention and compaction: delete removes old records according to time or size. compact retains the latest record for each key, subject to tombstone and offset cleanup behavior. Neither is a backup mechanism.
Production Operations
Monitor produce and fetch latency percentiles, under-replicated partitions, ISR shrinkage, request queue time, disk usage, GC pressure, network saturation, and page-cache eviction.
Size partitions for expected throughput and consumer concurrency. A partition cannot process records in parallel, while excessive partitions increase controller, broker, and file-descriptor overhead.
Partition leaders should be distributed across brokers to avoid concentrating traffic on one machine.
Leader elections, unavailable replicas, and an insufficient ISR can temporarily reject writes or weaken the configured durability guarantees.
Use TLS, SASL authentication, and topic ACLs in production, accounting for TLS’s additional CPU and copying costs.
During rolling upgrades, preserve the replication factor and min.insync.replicas.
Kafka replication protects against broker failure, not corruption or deletion. Use an external backup or replication system and regularly test restoration procedures.
64
How do you size an Apache Kafka cluster?
KnownRevise
Capacity-based Kafka cluster sizing starts with measured traffic, retention, replication, and availability requirements—not a fixed brokers-per-topic rule. The resulting design must sustain peak throughput, survive a broker failure without violating the SLA, and retain enough disk headroom for reassignments, compaction, and traffic spikes.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode
Sizing Inputs
Collect representative production data rather than relying only on averages:
Ingress rate: Peak messages per second and bytes per second for producers.
Message size: Include keys, headers, and value distributions; large records require fewer messages but more network, heap, and I/O.
Retention: Estimate both time-based retention and any retention.bytes limit.
Replication factor: Usually 3 for critical production topics.
Consumer behavior: Number of consumer groups, maximum parallelism, expected lag, and how quickly processing must catch up.
Availability target: Determine how many brokers, replicas, and KRaft controllers must remain available.
Compression and batching: compression.type, producer batch.size, and linger.ms materially affect throughput.
For compacted topics, size for churn, the active key set, and old record versions awaiting compaction rather than assuming data disappears immediately. Tombstones remain until compaction occurs and the topic’s delete.retention.ms window has elapsed; interrupted compaction can leave old records temporarily on disk.
Architecture and Partition Sizing
A broker stores topic partitions. Each partition has one leader and optional follower replicas; consumers in the same consumer group divide partitions among themselves. Therefore:
Partition count determines producer and consumer parallelism.
Ordering is guaranteed only within one partition.
A topic with 12 partitions and replication factor 3 creates 36 partition replicas.
Increasing partitions permits more consumers, but adds metadata, file handles, replication traffic, and rebalance work.
Choose the partition count from benchmarked per-partition throughput, not an arbitrary rule of thumb. A useful initial broker estimate is:
Benchmark with realistic record sizes, compression, acknowledgements, and replica placement. The design must still meet peak traffic after one broker fails. For high availability, use three KRaft controllers across failure domains; controllers should normally run on dedicated nodes even though they consume less CPU and memory than brokers.
Storage and Network Capacity
First calculate logical retained data:
retained bytes = messages per second × average uncompressed message bytes × retention seconds
Divide that requirement by the target maximum disk utilization. Keeping brokers around 60–70% utilized is a practical starting point, leaving space for segment creation, partition reassignment, temporary retention overlap, and traffic spikes.
Also budget:
Interbroker network: Replication and follower fetches, especially during reassignment or recovery.
Client network: Producer ingress and consumer egress; replicated storage does not multiply client ingress by the replication factor.
Request handling: Partition requests, compression CPU, authentication, and TLS overhead.
Internal topics: Consumer offsets, transaction state, and schema or Connect-related state.
Headroom: At least enough capacity to recover after a failure without exceeding disk safety limits.
Kafka Connect worker pools, Kafka Streams instances with state stores, and Schema Registry clusters have separate CPU, memory, disk, and availability requirements. Use dedicated infrastructure when their workload could destabilize the broker cluster.
Code Example
Create and inspect a replicated topic with a durable ISR threshold:
The 12 partitions must be validated against measured throughput and consumer concurrency. min.insync.replicas=2 with replication factor 3 allows writes while one replica is unavailable, assuming at least two replicas remain in the ISR.
Production Failure Modes and Operations
Delivery semantics: acks=all with enable.idempotence=true protects acknowledged writes from producer retry duplicates. Consumer-side at-least-once processing can still duplicate work if a consumer commits its offset before processing completes.
Exactly-once processing: Use Kafka transactions and isolation.level=read_committed, or Kafka Streams processing.guarantee=exactly_once. External databases require transactional integration or idempotent consumers.
Monitoring: Alert on under-replicated and offline partitions, ISR changes, consumer lag, request latency, broker disk utilization, GC pauses, network saturation, and producer errors.
Broker failure: With three replicas and two required ISR replicas, one broker loss should not stop writes, but leader elections and re-replication still create temporary latency.
Partition growth: Use bin/kafka-topics.sh --alter --topic <topic> --partitions <new-count> to increase partitions; use kafka-reassign-partitions.sh to redistribute partitions or replicas in a controlled operation.
Upgrades: Follow Kafka’s version-specific rolling-upgrade procedure, maintain controller quorum, and verify replication health after each stage.
Disaster recovery: Replication is not a backup. Use a geographically separate cluster, typically with MirrorMaker 2, and test recovery objectives.
Security: Use TLS in transit, SASL authentication, least-privilege ACLs, encrypted disks, and controlled certificate rotation. Place brokers across separate failure domains and avoid exposing plaintext listeners outside the trusted network.
65
How does Apache Kafka redistribute partition replicas between brokers?
KnownRevise
Apache Kafka redistributes partition replicas through partition reassignment: the controller records a new replica assignment, new replicas copy the partition’s log until they catch up, and obsolete replicas are removed. This is separate from a consumer group rebalance, which assigns topic partitions to consumer instances.
Read full answerHide full answer
Applies toApache Kafka 3.6–4.x in KRaft mode
Core Concept
A Kafka partition has one leader and usually one or more follower replicas. The leader serves client requests, while followers maintain copies for fault tolerance and potentially read scaling.
Kafka does not automatically move existing partition replicas when brokers are added or removed. An operator requests reassignment when decommissioning a broker, expanding capacity, changing rack placement, repairing replica placement, or redistributing disk load.
Operation
What changes
Typical trigger
Partition reassignment
Replica placement and data location
Broker replacement or capacity rebalancing
Leader election
Which replica is the leader
Preferred-leader election or broker failure
Consumer group rebalance
Which consumers own partitions
Consumer membership or partition-assignment changes
The logical partition and its records remain unchanged during reassignment. For example, orders-3 continues to contain the same records; only the brokers storing its replicas change.
Under-the-Hood Sequence
The controller validates the requested broker IDs and replica-assignment constraints.
The new assignment is recorded in the KRaft metadata log.
The Kafka replica fetcher copies records from the current leader, or from a subsequently elected leader if leadership changes.
The new replica catches up to the leader’s log-end offset and joins the in-sync replica set, or ISR.
Once the replacement is caught up, the obsolete replica is removed from the partition’s replica set.
An operator can optionally request a preferred-leader election after the target leader is in the ISR. The preferred leader is the first broker in the partition’s current replica list.
Replica movement is asynchronous. During the transition, a partition can temporarily contain more replicas than its configured replication factor because Kafka adds the replacement before removing the obsolete replica. Replication traffic can affect client latency, so leader and follower replication throttles can constrain migration speed.
Operational Workflow
The administrative workflow is implemented by kafka-reassign-partitions.sh. The operator supplies a version 1 assignment document containing the topic, partition, target replica broker IDs, and target log directories, then applies it with the tool’s execution mode.
The controller expands the replica set and waits for the new replica to catch up before contracting the set. Administrators inspect the partition with kafka-topics.sh --describe and verify the reassignment with the verification mode of kafka-reassign-partitions.sh. A preferred-leader change can then be requested with kafka-leader-election.sh when required.
Failure Modes and Monitoring
Insufficient destination capacity: The new replica cannot catch up, so reassignment remains incomplete.
ISR degradation: If too many replicas leave the ISR, writes can fail when min.insync.replicas cannot be satisfied.
Data loss risk: Keep unclean.leader.election.enable=false; an out-of-sync replica must not be promoted automatically.
Leader movement: Leadership changes can cause transient request retries or brief unavailability.
Consumer impact: Consumers normally retain the same partition assignments. Replica movement can cause metadata updates and transient retries, but it does not by itself require a consumer-group rebalance.
Monitoring: Track UnderReplicatedPartitions, UnderMinIsrPartitionCount, OfflineReplicaCount, leader-change rate, request latency, disk utilization, replication throughput, and consumer lag.
Production Best Practices
Preserve redundancy: For critical topics, commonly use replication.factor=3 and min.insync.replicas=2, adjusted to the required failure-domain tolerance.
Plan capacity: Ensure destination brokers have enough disk space and network capacity in their configured log.dirs.
Use rack awareness: Set broker.rack and distribute replicas across failure domains.
Throttle migrations: Use leader.replication.throttled.replicas and follower.replication.throttled.replicas carefully, because these broker-level settings can affect multiple partitions.
Decommission safely: Reassign partitions, verify completion and ISR health, and only then stop the broker.
Maintain independent backups: Replication does not protect against accidental deletion, malformed writes, or retention expiry; use cross-cluster replication or snapshots when independent recovery is required.
Secure administration: Use TLS and SASL, restrict administrative access, and provide protected client properties to administrative tools with --command-config.
66
How can an Apache Kafka cluster be upgraded without downtime?
KnownRevise
A no-downtime Kafka upgrade uses a rolling upgrade: replace one node at a time while preserving replication, KRaft quorum, and client connectivity. The cluster remains available, although leader elections can cause short latency increases and retriable client errors.
Read full answerHide full answer
Applies toApache Kafka 4.1.x with KRaft; rolling paths are 4.0 to 4.1 and 3.9 to 4.0 to 4.1
Upgrade Strategy
Upgrade one node at a time: Never restart multiple brokers or controllers simultaneously. Keep enough replicas and voting controllers online to preserve quorum and partition leadership.
Upgrade KRaft controllers first: Upgrade the controller quorum one controller at a time, then upgrade broker-only nodes. For combined broker/controller nodes, upgrade them individually while retaining at least two of three voters.
Follow the compatibility path: Upgrade directly only across officially supported versions. A 3.9 cluster must first be upgraded to 4.0 before proceeding to 4.1.
Leave ZooKeeper clusters first: Kafka 4.x does not support ZooKeeper. A ZooKeeper-based cluster must be migrated to KRaft using a 3.9 release before it can enter the Kafka 4.x rolling-upgrade path.
Upgrade clients afterward: Keep existing clients running during the broker rollout, then upgrade Connect workers, Kafka Streams applications, Schema Registry instances, and other clients gradually.
Preserve rollback compatibility: Do not advance the message-format version or enable new KRaft metadata features until every node is upgraded and the rollback window has closed.
Pre-Upgrade Safeguards
Area
Recommended condition
Controllers
Three or more KRaft voters; restart only one at a time
At least 50% free disk and sufficient network headroom
Recovery
Metadata quorum snapshot, configuration and ACL backups, plus replicated partition data
Before starting, confirm that there are no under-replicated or offline partitions, disk errors, or abnormal consumer lag. Use kafka-broker-api-versions.sh to inspect broker-supported protocol versions; obtain connected client versions from application inventory or client metrics instead.
Rolling Upgrade Procedure
Record the current binary version, broker configuration, KRaft node.id, listeners, security settings, and voter configuration such as controller.quorum.voters for static quorums.
Create a consistent KRaft metadata snapshot using kafka-metadata-quorum.sh snapshot; also back up broker and topic configurations and ACLs. Use a secondary cluster or equivalent replication for partition-data disaster recovery rather than relying on a metadata snapshot alone.
Stop one node cleanly and install the target Kafka release while preserving its data directories and configuration. Do not reformat an existing KRaft storage directory.
Ensure controlled.shutdown.enable=true so leadership can be moved before shutdown.
Point the service definition at the new installation, start the upgraded node, and wait for it to rejoin normally.
Verify controller quorum, partition leadership, ISR status, consumer lag, broker health, and client error rates before upgrading another node.
Upgrade the KRaft controllers first, one at a time, and then broker-only nodes. If roles are combined, maintain controller quorum throughout.
After all brokers are upgraded, roll Connect workers, Kafka Streams applications, Schema Registry instances, and clients gradually, validating each stage.
Validation and Client Behavior
acks=all with min.insync.replicas=2 prevents acknowledged writes from being accepted with weaker replication; unavailable writes may block instead.
Ordering is guaranteed only within a partition.
Producer idempotence prevents duplicate records from producer retries within the producer’s retry and session guarantees.
Kafka read-process-write exactly-once processing requires a transactional producer with a unique transactional.id, enable.idempotence=true, and consumers using isolation.level=read_committed. Kafka Streams additionally uses processing.guarantee=exactly_once. This does not make arbitrary external side effects exactly-once.
Consumers should use bounded exponential backoff for retriable errors such as NOT_LEADER_OR_FOLLOWER and REQUEST_TIMED_OUT.
Code Example
Run these commands from a host containing the target Kafka distribution:
BROKER=kafka-1.example.com
BROKER_ID=1
BROKER_SERVICE=kafka.service
# Check the current KRaft quorum state.
./bin/kafka-metadata-quorum.sh \
--bootstrap-server "$BROKER:9092" describe --status
# List under-replicated partitions.
./bin/kafka-topics.sh \
--bootstrap-server "$BROKER:9092" \
--describe --under-replicated-partitions
# Inspect consumer-group assignments and lag.
./bin/kafka-consumer-groups.sh \
--bootstrap-server "$BROKER:9092" \
--describe --group payments
# Inspect APIs supported by the brokers; this does not report client versions.
./bin/kafka-broker-api-versions.sh \
--bootstrap-server "$BROKER:9092"
# Save the broker's dynamic configuration.
./bin/kafka-configs.sh \
--bootstrap-server "$BROKER:9092" \
--describe --entity-type brokers --entity-name "$BROKER_ID" --all
# Restart only this broker after pointing the unit at the new installation.
sudo systemctl restart "$BROKER_SERVICE"
Repeat the validation commands after each restart and before proceeding.
Failure Modes and Rollback
Reduced redundancy: Stopping one node removes a replica and may trigger leader elections. Stop if remaining replicas cannot satisfy min.insync.replicas.
Quorum loss: Restarting two of three controllers simultaneously can make the cluster unavailable.
Disk pressure: Log growth, compaction, and replica lag can consume space. Monitor log-dir-size, I/O latency, and request queues.
Incompatible metadata: Do not assume binary downgrade is safe after advancing message-format or KRaft metadata features. Roll back before such changes when the release documentation permits it.
Security regressions: Preserve TLS and SASL listener configuration, rotate certificates beforehand, and ensure upgraded broker identities remain valid.
Connector disruption: Connect worker replacement can restart tasks and replay records. Validate sink idempotency and monitor duplicate handling.
“No downtime” means continuous service availability, not zero request latency. Capacity planning, redundancy, staged validation, and tested rollback procedures are essential.
67
How does MirrorMaker 2 replicate Kafka data across clusters?
KnownRevise
MirrorMaker 2 is Apache Kafka’s replication system for moving records between independent Kafka clusters. It runs on Kafka Connect, preserves ordering within each source partition, and normally provides at-least-once replication for disaster recovery and active-active use cases.
Read full answerHide full answer
Applies toApache Kafka 3.9 and 4.x
Architecture and Flow
MirrorMaker 2 uses a distributed Connect worker and the MirrorSourceConnector:
Source replication: The connector consumes records from source-cluster topics, creates corresponding target topics when necessary, and produces the records to the target cluster.
Heartbeat connector: MirrorHeartbeatConnector emits heartbeat records used for cross-cluster lag measurement and replication-cycle detection.
Checkpoint connector: MirrorCheckpointConnector stores source topic offsets in a target-cluster checkpoint topic so replication can resume after a restart.
Offset-sync connector: MirrorOffsetSyncConnector optionally synchronizes consumer-group offsets. The offset-sync topic can be located in either the source or target cluster.
For example, orders.created in cluster A is replicated to orders.created in cluster B. The default replication policy normally preserves the source partition count and maps a source partition to the corresponding target partition.
Model
Behavior
Operational trade-off
Active-passive
Cluster B receives replicated data but normally accepts no application writes until failover
Simplest recovery model; consumers must be redirected during promotion
Active-active
Applications write to both clusters
Requires strict topic, consumer-group, and loop-prevention policies; cross-cluster ordering cannot be guaranteed
Kafka 4.x clusters use KRaft, but MirrorMaker 2 communicates through Kafka client and data APIs and does not directly access the controller quorum.
Replication and Consumer-Group Behavior
Ordering: Records are produced in source-partition order to the mapped target partition. There is no global ordering across partitions or topics.
Delivery: Replication is at-least-once. A failure after the target accepts records but before the source connector commits its offset can cause duplicates, so consumers should be idempotent or use deduplication.
Exactly once: MirrorMaker 2 does not provide transactional exactly-once processing across independent clusters by itself. End-to-end exactly-once behavior requires an application-specific transactional and deduplication design.
Consumer-group failover: With offset synchronization enabled, consumer-group offsets can be made available for use in the secondary cluster. During failover, stop writes to the old active cluster, confirm that replication and offsets are current, and then redirect consumers to the promoted cluster.
Active-active caution: Heartbeats prevent immediate feedback loops, but application writes can still create logical loops. Use coordinated topic mappings and separate application group identifiers where appropriate.
Key Configuration
Important worker and connector options include:
source.cluster.alias and target.cluster.alias: Identify the clusters configured on the Connect worker.
connector.class: Set to org.apache.kafka.connect.mirror.MirrorSourceConnector.
topics: Regular expression selecting the topics to replicate.
replication.policy.class: Selects the policy governing topic and partition mapping and replication-cycle handling.
offset-syncs.topic.location: Selects the cluster in which synchronized consumer offsets are stored.
sync-group.connector.groups: Restricts synchronization to matching consumer-group names.
heartbeat.interval.ms: Controls heartbeat frequency and therefore lag-detection resolution.
MirrorMaker 2 replicates records, not other cluster metadata. Kafka Streams state stores, Schema Registry schemas, ACLs, and topic configuration require separate management. Consumer-group offsets can optionally be synchronized, but group assignments, in-flight work, and other consumer state are not replicated. The target log is compacted normally according to the target topic’s own compaction configuration.
Failure Modes and Security
Source unavailable: The connector remains retryable and resumes from its last committed position when source data becomes available again, subject to source retention.
Target unavailable: Producers retry until delivery limits are reached, after which the Connect task fails and the framework retries it. Monitor producer errors and lag; do not delete checkpoint topics as a recovery mechanism.
Consumer lag: A replicated target can still have high consumer lag because replication throughput does not guarantee downstream processing capacity.
Data loss or corruption: Replication is not a complete backup. It can propagate deletions or corruption, and recovery depends on target retention and integrity.
Security: Use TLS and SASL for cluster connections. The source identity needs read access to replicated topics and permissions needed by offset synchronization; the target identity needs write access to replicated records and permission to create or write heartbeat and checkpoint topics. If offset-sync storage is enabled in the source, that cluster also needs the corresponding topic permissions.
Connect REST access: Restrict connector creation and configuration changes to administrators, and expose the REST API only through appropriate network controls and authentication.
Production Practices
Monitor connector and task state, retries, producer errors, heartbeat-based replication lag, target consumer lag, broker latency, disk usage, under-replicated partitions, and leader elections. Back up both Kafka clusters and the Connect configuration, offset-storage, and status-storage topics using appropriate Kafka backup procedures.
Upgrade the source and target brokers according to Kafka’s supported rolling-upgrade sequence, then upgrade the Connect workers running MirrorMaker 2 and verify connector status after each worker change. Because MirrorSourceConnector is single-task, scale large workloads by increasing source-topic partition counts or deploying separate connector instances with disjoint topic selections rather than relying on tasks.max alone. Ensure target brokers, retention, and consumer capacity can absorb the replication peak during failover.
68
How do you design Apache Kafka for disaster recovery?
KnownRevise
A resilient Kafka disaster-recovery design combines multi-AZ replication, independent backups or a secondary region, and explicit RPO/RTO targets. Broker replication survives hardware and zone failures, but not regional failure, deletion, corruption, or ransomware.
Read full answerHide full answer
Applies toApache Kafka 4.x with KRaft and confluent-kafka 2.x
Failure Model and Recovery Objectives
Failure
Primary protection
Effect
Broker or disk failure
In-sync replicas
Automatic failover; acknowledged data survives if enough replicas remain
Availability-zone failure
RF 3 across three zones
Service continues with reduced capacity
Region failure
MirrorMaker 2, dual writes, or another independent copy
Fail over or redirect clients
Deletion or corruption
Immutable, versioned backups
Restore selected topics or the cluster
Ransomware or credential compromise
Separate credentials and immutable backups
Recover into a clean environment
RPO is the maximum acceptable data loss; RTO is the maximum acceptable recovery time. Multi-region replication normally has an RPO equal to observed replication lag, not zero.
Data Durability Architecture
Run at least three brokers across three zones. In combined mode, spread a three-controller quorum across zones; in dedicated mode, run at least three controllers separately from the brokers.
Use replication factor 3 and min.insync.replicas=2 for critical topics. Keep unclean.leader.election.enable=false to prevent data loss during an unclean election.
Configure producers with acks=all and enable.idempotence=true. Idempotence prevents duplicates from producer retries, but it is not a backup.
Kafka preserves ordering only within a partition. At-least-once consumers must process a record before committing its offset.
Kafka’s exactly-once read-process-write semantics for Kafka records require transactional producers, a stable transactional.id, isolation.level=read_committed consumers, and a suitably replicated transaction-state log. External side effects still require idempotency or application-specific coordination.
Protect Kafka Streams state-store changelogs and application offsets. Kafka Connect configurations and sink idempotency must also be considered because sink writes can be repeated.
Backup and Restore
Back up broker data directories, metadata.log.dir if separate, configuration, ACLs, and the original cluster ID. Store encrypted, immutable copies with separate credentials. Do not copy live broker files without a supported snapshot or consistency mechanism.
For a new KRaft cluster, initialize every node with the same cluster ID and node-specific configuration containing a unique node.id:
CLUSTER_ID=$(./bin/kafka-storage.sh random-uuid)
./bin/kafka-storage.sh format \
--cluster-id "$CLUSTER_ID" \
--config server.properties
During restoration, restore data and metadata consistently. Do not re-format restored metadata or change the cluster ID.
A recovery sequence is:
Restore brokers into an isolated network.
Start a controller quorum and verify it with kafka-metadata-quorum.sh.
Restore remaining brokers gradually and check partition leadership.
Validate ISR, topic metadata, consumer offsets, and representative records.
Reconnect applications only after completeness checks pass.
Test restores regularly. An untested backup is only an assumption.
Cross-Region Recovery
MirrorMaker 2: Asynchronously replicates topics and consumer offsets. Failover requires changing bootstrap addresses or routing outside Kafka.
Active-active dual writes: Publish equivalent records to both regions. This reduces lag-related loss but requires idempotent consumers or reconciliation for partial writes.
Replicate Schema Registry schemas and compatibility policies separately; topic replication does not copy Schema Registry metadata.
Region-specific consumer-group IDs simplify independent operations. Reusing IDs across independently operated clusters requires an explicit offset-replication and failover policy.
Test bandwidth, encryption, replication lag, partial writes, and ordering under production load.
Code Example
Create a replicated topic and explicitly enforce its durability settings:
A Python producer using idempotence and acknowledgements from all in-sync replicas:
from confluent_kafka import Producer
def delivery_report(err, msg):
if err is not None:
print(f"Delivery failed: {err}")
else:
print(f"Delivered to {msg.topic()}[{msg.partition()}]")
producer = Producer({
"bootstrap.servers": "kafka.example.com:9092",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "SCRAM-SHA-512",
"sasl.username": "orders-producer",
"sasl.password": "password",
"enable.idempotence": True,
"acks": "all",
})
producer.produce(
"orders",
key=b"order-123",
value=b'{"status":"paid"}',
on_delivery=delivery_report,
)
producer.flush(10)
Production Practices
Monitor under-replicated and offline partitions, leader elections, disk usage, request latency, consumer lag, replication lag, backup age, and restore-test age.
Use TLS, SASL, least-privilege ACLs, administrative MFA, secret rotation, and separate backup credentials.
Upgrade brokers and clients in rolling stages while watching ISR, latency, and error rates. Keep recovery procedures compatible with the Kafka version.
Exercise broker loss, zone loss, regional failover, offset restoration, and backup restoration through game days.
Automate infrastructure while treating Kafka metadata and cluster identity as controlled recovery artifacts.
69
What is Apache Kafka tiered storage?
KnownRevise
Apache Kafka tiered storage keeps hot partition segments on the broker’s local disks and moves older, closed segments to a remote object store, while producers, consumers and consumer groups keep using the same APIs, the same offsets and the same replication model.
Read full answerHide full answer
Applies toApache Kafka 4.x (tiered storage was early access in 3.6 and became production-ready in 3.9) with a plugin implementing the Kafka remote-storage SPIs
How It Works
Kafka partitions are divided into log segments. During normal operation:
Producers append new records to the active segment in the partition leader’s local tier.
When a segment rolls over, the broker uploads it to the remote tier through a remote-storage plugin that implements the RemoteStorageManager and RemoteLogMetadataManager SPIs.
The broker records remote-log metadata for the segment — its offsets, timestamps, size and remote location — in the metadata manager.
After the upload is confirmed, the local copy is kept only until it falls outside the topic’s local.retention.ms / local.retention.bytes; then it is deleted locally, while overall retention.ms / retention.bytes govern the remote copy.
Consumers keep using the same partition and offset. When a consumer needs data that is no longer local, the broker fetches it from the remote tier and streams it to the client; remote data is not written back into the local log, though the remote-segment index files are still stored locally.
Tier
Typical contents
Trade-off
Local tier
Active segments, indexes, remote-log index files
Low latency and high throughput, but expensive capacity
Remote tier
Older closed segments
Lower storage cost, but higher read latency and network dependency
Tiered storage is not a replacement for replication: normal leader/follower replication still applies, and the plugin is responsible for safely uploading, retrieving and deleting remote segments.
Enabling It
Apache Kafka ships the framework, but no official object-store implementation; you must supply a plugin whose classes implement the remote-log SPIs.
On each broker that will host partitions of the topic, set remote.log.storage.system.enable=true in the broker properties. This is a read-only broker setting, so it requires a broker restart. Point remote.log.storage.manager.class.name (and, if you do not use the built-in topic-based metadata manager, remote.log.metadata.manager.class.name) at the classes provided by your plugin, and make sure the plugin jars are on the class path of every broker involved.
On the topic, set the topic-level config remote.storage.enable=true — either at creation time with the --config option of kafka-topics.sh, or afterwards with kafka-configs.sh using --alter --add-config against the topic entity. Confirm it with kafka-configs.sh --describe for the topic. Before Kafka 3.9 tiered storage could not be turned off for a topic; from 3.9 it can be disabled, but doing so requires remote.log.delete.on.disable=true, which deletes the topic’s remote data, so plan the data lifecycle before enabling it. Compacted topics cannot use tiered storage. Enabling it on an existing topic means the broker will upload its existing closed segments, so plan for that I/O.
Consumer and Delivery Semantics
Tiered storage does not change Kafka’s fundamental guarantees:
Ordering: records are ordered within a partition, not across partitions.
At-least-once delivery: consumers commit offsets after processing and may reprocess records after a failure.
Exactly-once processing: requires Kafka transactions, or the appropriate idempotent producer and read-committed consumer configuration; it is not conferred by moving data to remote storage.
Consumer groups: offsets remain managed by the consumer-group protocol; internal topics such as __consumer_offsets are not tiered by this feature.
Latency: a remote read adds fetch latency on cache misses, so consumers reading old data may see slower fetches and temporarily higher lag.
Production Operations and Failure Modes
Upload failures: investigate plugin errors, object-store throttling, expired credentials and network reachability; monitor upload lag and the progress of segment checkpoints.
Remote read failures: track fetch latency, error rate and consumer lag; a broker must be able to reach the object store to serve remote segments.
Local disk pressure: monitor broker disk usage including active segments, the local cache of remote-log index files (sized by remote.log.index.file.cache.total.size.bytes).
Data durability: use object-store versioning, provider-side replication, lifecycle policies and an independent backup strategy where the retention requirement demands it.
Metadata recovery: protect KRaft metadata separately from the remote segment data; restoring objects without the matching Kafka metadata and topic configs can leave partitions unreadable.
Upgrades: upgrade broker and plugin together, check SPI compatibility, and rehearse remote reads, leader failover, broker replacement and cleanup before rolling out.
Security and Trade-offs
Use TLS and SASL for client-to-broker traffic, and secure broker-to-object-store traffic with private endpoints, IAM roles or narrowly scoped credentials. Never put storage credentials in topic configuration or on command lines visible in shell history and logs; rely on the provider’s encryption controls for data at rest.
The benefit is lower storage cost and better use of local disks. The costs are remote-fetch latency, a hard dependency on the storage provider, extra operational complexity and possible cleanup or consistency issues in the plugin. For latency-sensitive topics, keep hot data local longer and use tiered storage mainly for segments that are read rarely.
Best Practices and Common Pitfalls
Start with a non-critical topic and measure upload throughput, remote-read latency and local-disk savings before rolling out.
Size the local tier for active data, indexes and compaction work, not just active partitions.
Test leader failover and broker replacement while remote segments are being read.
Ensure the plugin implements durable deletion and retention cleanup correctly; orphaned objects cost money and complicate recovery.
Do not assume tiering changes consumer offsets, transaction semantics, partition ordering or replication guarantees.
Keep a tested restore procedure covering object-store data, Kafka metadata, topic configs and plugin configuration.
70
Self-managed Kafka vs Confluent Cloud, Amazon MSK, and Redpanda
KnownRevise
Self-managed Apache Kafka provides maximum control over infrastructure, deployment, upgrades, and cost, while managed Kafka services offload broker operations. Confluent Cloud and Amazon MSK are managed Kafka offerings; Redpanda is a Kafka-API-compatible alternative with a different internal architecture and ecosystem coverage.
Read full answerHide full answer
Applies toApache Kafka 4.x; current Confluent Cloud, Amazon MSK, and Redpanda managed offerings
Architecture and Ecosystem
Area
Self-managed Kafka
Confluent Cloud
Amazon MSK
Redpanda
Deployment
VMs, containers, Kubernetes with Strimzi, or bare metal
Vendor-operated Kafka clusters
AWS-managed provisioned or serverless clusters
Single-binary, Kafka-compatible service using Raft
Control plane
KRaft controllers in current Kafka 4.x deployments
Fully vendor-managed
Managed; KRaft is available for supported provisioned configurations
Redpanda’s Raft implementation, not Kafka KRaft
Availability
Designed by the customer
Region and tier dependent
AWS availability-zone and deployment dependent
Vendor-managed regions and cluster configurations
Kafka Connect
Full Kafka distribution support
Native Confluent Connect
MSK Connect with supported source and sink connectors
Redpanda Connect; connector and configuration compatibility must be validated
Kafka Streams
Supported by Apache Kafka
Supported and managed by Confluent
Supported, subject to client compatibility
Kafka API compatibility exists, but support and feature parity are not identical to Apache Kafka
Schema Registry
Separate Confluent Schema Registry deployment
Integrated managed service
Usually integrated through supported registry or external tooling
Built into the Redpanda broker binary (Confluent-compatible API)
Operations
Customer provisions, monitors, patches, scales, and upgrades brokers
Vendor handles broker lifecycle and most infrastructure work
AWS handles broker lifecycle and infrastructure work
Vendor handles the managed service; customers still tune topics, clients, and access
Self-managed Kafka is not automatically cheaper or more reliable. Its flexibility increases operational responsibility, especially when managing broker failures, disk capacity, controller quorum, network isolation, and rolling upgrades.
Delivery Guarantees
Managed services do not change Kafka’s fundamental delivery model:
Ordering: Records are ordered within a partition, not across an entire topic. A key routed consistently to one partition preserves ordering for that key.
At-most-once delivery: Producers avoid retries and acknowledge processing before persistence. It can lose records after a producer failure.
At-least-once delivery: Producers retry failed sends, and consumers commit offsets after processing. Duplicate records are possible, so handlers should be idempotent.
Exactly-once processing: Kafka’s transactional model can atomically publish records, consume input records, and commit consumer offsets within Kafka.
For transactional Kafka workloads, use enable.idempotence=true, acks=all, a unique stable transactional.id per producer instance, transactional send-and-offset APIs, and isolation.level=read_committed on consumers. Merely enabling idempotence does not make external database writes exactly-once; use an outbox pattern, idempotent writes, or another coordination mechanism.
Production Operations
For self-managed Kafka, monitor broker disk utilization, request latency, network throughput, under-replicated and offline partitions, controller health, JVM pressure, and consumer lag. An exact topic inspection command in a standard Kafka installation is:
Capacity planning: Add brokers before disks fill. A partition is assigned to one broker, so partition count affects rebalance and recovery time.
Hot partitions: A popular key can overload one partition or broker. Review partition-level traffic rather than only cluster averages.
Consumer lag: Lag can indicate slow processing, a rebalance, a connectivity problem, or an undersized consumer group.
Rebalances: Long processing cycles can exceed max.poll.interval.ms and cause repeated partition revocation.
Configuration: min.insync.replicas=2, replication factor 3, and acks=all commonly prevent acknowledged records from being lost with one broker failure. This reduces write availability during larger failures.
Retention: retention.ms controls time retention, while cleanup.policy=compact retains the latest value for each record key. Compaction does not immediately remove every older duplicate.
Upgrades: Test client, broker, Connect, and Schema Registry compatibility. Upgrade brokers in a controlled rolling sequence and verify controller quorum health.
Managed platforms remove broker patching and infrastructure failures from the customer’s responsibility, but they do not remove the need to monitor lag, errors, quotas, partition skew, schema compatibility, and client configuration.
Security and Failure Modes
Use TLS and authenticated protocols such as SASL/SCRAM or SASL/OAuth where supported, restrict broker access with network controls, and apply least-privilege ACLs or role-based access. Managed services may provide IAM integration, private networking, and key-management options, but application credentials and authorization design remain customer responsibilities.
Replication is not a backup: corruption, accidental deletion, or faulty producers can be replicated too. Define retention, cross-region recovery, and backup procedures explicitly. MirrorMaker 2 or a managed disaster-recovery product can protect against regional failure, but replication latency and operational complexity must be included in the recovery plan.
Selection Guidance
Choose self-managed Kafka when strict infrastructure control, custom networking, predictable high utilization, or existing Kafka expertise justify operating the platform. Choose Confluent Cloud when the broader Confluent ecosystem and vendor-managed operations matter most. Choose Amazon MSK for tight AWS integration, provisioned or serverless consumption models, and AWS networking and governance. Choose Redpanda or Redpanda Cloud when a simpler Kafka-compatible architecture is attractive, but validate connector, transaction, schema-management, and client compatibility before migration.
71
How do you test Kafka-based applications reliably?
KnownRevise
Integration testing against real Kafka brokers is the core of testing Kafka-based applications: layer it with unit tests, contract tests, failure injection and end-to-end tests so application logic and infrastructure-dependent behaviour are both covered.
Do not mock the Kafka protocol in component tests: mocks hide serialization, metadata refresh, rebalance, transaction and offset-management bugs.
Build an Isolated Test Environment
Run an ephemeral Kafka environment in CI with Testcontainers, Docker or an IaC-managed sandbox.
Kafka 4.x is KRaft-only: ZooKeeper metadata mode was removed in Kafka 4.0.
Give every test run a unique topic and consumer group to avoid offset and data contamination.
For production-like clusters use replication factor 3, min.insync.replicas=2 and unclean.leader.election.enable=false.
Set offsets.topic.replication.factor, transaction.state.log.replication.factor and transaction.state.log.min.isr when testing transactions.
Exercise Kafka Connect worker config, converters, error handling and connector plugins against the same broker.
Use TopologyTestDriver for fast Kafka Streams topology checks, then run the real Streams runtime in integration tests.
Validate Schema Registry compatibility modes and put schema evolution into contract tests.
Cover Delivery, Failure and Data Contracts
Ordering holds only within a partition, never across a topic; use a consistent partition key when order matters.
At-least-once delivery: acknowledge the broker and commit offsets only after successful processing; a crash between processing and commit yields duplicates.
Exactly-once semantics need idempotent producers, Kafka transactions and isolation.level=read_committed; they do not cover external databases or HTTP side effects, which need an outbox or idempotency key.
Test unavailable brokers, broker restarts, leader elections, rebalances, delayed responses, disk pressure and malformed events.
Use unique records and explicit timeouts so a failed test cannot consume another test’s messages.
Code Example
A pytest fixture creates an isolated topic and checks a producer-consumer round trip; in application tests, call your Kafka adapter instead of produce/consume directly.
import os
import time
import uuid
import pytest
from confluent_kafka import Consumer, Producer
from confluent_kafka.admin import AdminClient, NewTopic
BOOTSTRAP = os.getenv("KAFKA_BOOTSTRAP_SERVERS", "localhost:9092")
@pytest.fixture
def kafka_topic():
topic = f"orders-test-{uuid.uuid4().hex}"
admin = AdminClient({"bootstrap.servers": BOOTSTRAP})
future = admin.create_topics([
NewTopic(
topic,
num_partitions=3,
# RF=1 because the ephemeral test cluster is a single broker.
replication_factor=1,
config={"cleanup.policy": "delete"},
)
])[topic]
future.result(timeout=10)
yield topic
try:
admin.delete_topics([topic])[topic].result(timeout=10)
except Exception:
pass # Cleanup must not hide the original test failure.
def test_order_round_trip(kafka_topic):
producer = Producer({"bootstrap.servers": BOOTSTRAP})
delivery_errors = []
def on_delivery(error, message):
delivery_errors.append(error)
producer.produce(
kafka_topic,
key=b"order-1",
value=b'{"id": 42}',
on_delivery=on_delivery,
)
assert producer.flush(10) == 0
assert len(delivery_errors) == 1
assert delivery_errors[0] is None
consumer = Consumer({
"bootstrap.servers": BOOTSTRAP,
"group.id": f"test-{uuid.uuid4().hex}",
"auto.offset.reset": "earliest",
"enable.auto.commit": False,
})
try:
consumer.subscribe([kafka_topic])
deadline = time.monotonic() + 10
received = None
while time.monotonic() < deadline:
message = consumer.poll(1.0)
if message is None:
continue
if message.error():
raise RuntimeError(message.error())
received = message
break
assert received is not None
assert received.key() == b"order-1"
assert received.value() == b'{"id": 42}'
finally:
consumer.close()
A CLI smoke test checks broker and topic operations independently:
Treat the test cluster as untrusted: use TLS and SASL, least-privilege ACLs and per-environment credentials and topics.
Monitor consumer lag, under-replicated and offline partitions, request latency, broker disk usage and failed connector tasks.
Make tests deterministic with bounded timeouts, controlled fixtures and assertions on success and failure paths.
Test rolling upgrades and mixed-version compatibility before upgrading production brokers.
Replication is not a backup: verify mirrored clusters and restore procedures in separate DR drills.
Keep production brokers out of CI; use ephemeral or dedicated non-production clusters with production-like partition counts, replication, security and monitoring.
72
What are common Apache Kafka anti-patterns?
KnownRevise
Kafka anti-patterns are designs that look scalable or simple but cause data loss, duplicate processing, operational instability, or poor performance under production load. Avoiding them means matching partitioning, replication, consumer semantics, retention and monitoring to the system’s real failure model.
Read full answerHide full answer
Applies toApache Kafka 4.x (KRaft only — ZooKeeper was removed in 4.0)
Architecture and Topic Design
Anti-pattern
Production impact
Safer design
One topic per entity or tenant
Metadata, ACL and operational overhead grow fast
A bounded set of shared topics partitioned by a stable key
Too few partitions
Consumers cannot scale; hot keys bottleneck
Size partitions from throughput and concurrency needs
One-partition critical topic
No parallel consumption or partition failover
Multiple partitions with replication factor 3
Oversized or tiny records
Large records pressure memory and network; tiny ones waste overhead
Keep records moderate and batch where useful
Keyless or random partitioning
Related events can be reordered
A stable key routes related records to one partition
Hot partition: a popular key (one customer) sends all traffic to one partition, capping consumer parallelism and making that partition a single point of failure.
Changing partition counts carelessly: adding partitions re-maps keyed records, so stateful consumers can see incompatible assignments; settle the key strategy first.
Replication treated as backup: replicas protect availability, not logic errors; a bad producer’s records replicate everywhere, so keep independent backups.
Unsafe quorum settings: replication factor 3 with min.insync.replicas=2 tolerates one broker failure and rejects writes lacking quorum; leave unclean.leader.election.enable=false unless forced availability matters more.
Fragile KRaft deployments: run an odd-sized controller quorum (normally three controllers) across failure domains, and separate controller and broker resources where availability demands it.
Producers, Consumers, and Guarantees
Non-idempotent producers: a retried produce after a network failure can duplicate records; set enable.idempotence=true with acks=all and a nonzero retries, remembering idempotence covers the producer session, not application resubmits.
Wrong ordering assumptions: ordering is guaranteed per partition only; same-key records must land in the same partition and be consumed in order.
Committing offsets before processing: this loses events permanently; with enable.auto.commit=false, commit offsets only after successful processing (at-least-once).
Processing inside the poll loop: long work blocks poll() and triggers max.poll.interval.ms eviction and rebalance storms; poll often and use a bounded worker pool.
Ignoring rebalances: deployments, partition changes and slow consumers cause churn; use cooperative sticky assignment and keep processing bounded.
Assuming exactly-once everywhere: exactly-once needs a Kafka transaction (sendOffsetsToTransaction in the Java client, send_offsets_to_transaction in confluent-kafka) plus isolation.level=read_committed consumers; external databases and APIs remain at-least-once unless they join the transaction.
Unbounded retries or timeouts: tune delivery.timeout.ms and request.timeout.ms so broker failure surfaces before the application’s own deadline.
Connect, Streams, and Schemas
Excessive Connect concurrency: a high tasks.max can overload the sink, exhaust connection limits and add transactional overhead; start conservative and scale from measured sink capacity.
Assuming every connector is exactly-once: check the connector’s delivery guarantee and the sink’s idempotence; repeated writes are otherwise possible.
Unbounded Streams state: stateful operators can exhaust local disk and memory; watch state-store size, changelog lag and restore time, and repartition when key semantics change.
Ignoring Streams transactions: processing.guarantee=exactly_once_v2 needs transactional internal topics and sufficient replication; it still excludes arbitrary external side effects.
Schema Registry without compatibility checks: disabling compatibility breaks consumers during record evolution; enforce a compatibility mode per environment plus a subject naming strategy.
Production Operations and Security
No lag or replication monitoring: watch group lag, under-replicated and offline partitions, ISR churn, request latency, broker disk usage, GC pauses, Connect task failures and Streams thread lag.
Alerting on lag alone: lag spikes can be normal; combine lag with processing rate, age of the oldest unprocessed record and error rate.
Unplanned upgrades: follow the supported upgrade path, roll brokers one at a time, and watch rebalances, ISR movement and latency after each step.
Plaintext or over-broad access: use TLS for client and inter-broker traffic, SASL/SCRAM for authentication and topic ACLs for least privilege; protect admin credentials and audit ACL changes.
Missing recovery procedures: document broker replacement, controller recovery, topic restore, offset-reset decisions and data validation; back up Schema Registry separately, since topic replication does not protect it.
Guardrails
Verify the real cluster with kafka-topics.sh --describe, kafka-consumer-groups.sh --describe and kafka-configs.sh --describe instead of trusting application metrics alone, and gate topic creation, replication settings, producer retry behaviour, commit timing, schema compatibility, capacity limits, security configuration and tested recovery drills through review.
73
How do I write a Kafka producer and consumer with confluent-kafka?
KnownRevise
confluent-kafka is Kafka’s Python client, backed by librdkafka. A producer sends serialized records to topic partitions, while a consumer polls those partitions as part of a consumer group and can commit offsets after processing for at-least-once delivery.
Read full answerHide full answer
Applies toconfluent-kafka 2.x with Apache Kafka 3.x and 4.x brokers
Architecture and Client Behavior
Brokers: Kafka 4.x requires KRaft for controller quorum and metadata management. Kafka 2.8 introduced KRaft as an early-access preview, and Kafka 3.3 made it production-ready.
Partitions: Each partition is an ordered log. Partitioning allows a topic’s throughput to scale across brokers.
Replication: Partition leaders replicate records to followers. For important topics, use at least three brokers and a replication factor of three.
Consumer groups: A group assigns each partition to at most one consumer. Different groups consume the records independently.
Offsets: Consumer progress is stored in Kafka’s internal __consumer_offsets topic, which uses log compaction.
Partition selection: With no explicit partition, librdkafka uses the configured partitioner, normally deriving the partition from a non-null key. Increasing a topic’s partition count can change that mapping.
Delivery Semantics
Semantic
Required behavior
At-most-once
Commit before processing; a failure can lose unprocessed records.
At-least-once
Process successfully before committing; a failure between those steps can cause duplicates.
Exactly-once
Use a transactional producer, isolation.level=read_committed, disabled automatic offset commits, and atomically publish output and consumed offsets with send_offsets_to_transaction().
Ordering is guaranteed only within a partition. Idempotence prevents duplicates caused by producer retries, but not application-level resending after a restart. Exactly-once workflows use init_transactions(), begin_transaction(), send_offsets_to_transaction(), and commit_transaction(). A transactional producer requires a stable, unique transactional.id; conflicting instances are fenced.
Code Example
Install the client and create a topic. The replication factor below requires three brokers:
The following program contains both a producer and consumer:
# kafka_app.py
import argparse
import json
import os
from confluent_kafka import Consumer, KafkaError, KafkaException, Producer
def make_config(client_id, **extra):
config = {
"bootstrap.servers": os.environ["KAFKA_BOOTSTRAP_SERVERS"],
"security.protocol": os.getenv("KAFKA_SECURITY_PROTOCOL", "PLAINTEXT"),
"client.id": client_id,
}
config.update(extra)
if config["security.protocol"].startswith("SASL"):
config.update(
{
"sasl.mechanism": os.getenv(
"KAFKA_SASL_MECHANISM", "SCRAM-SHA-512"
),
"sasl.username": os.environ["KAFKA_USERNAME"],
"sasl.password": os.environ["KAFKA_PASSWORD"],
}
)
ca_location = os.getenv("KAFKA_SSL_CA_LOCATION")
if ca_location:
config["ssl.ca.location"] = ca_location
return config
def produce():
failures = []
def on_delivery(error, message):
if error is not None:
failures.append(str(error))
return
key = message.key()
print(
f"Delivered {key.decode() if key else '<null>'} to "
f"{message.topic()}[{message.partition()}] "
f"at offset {message.offset()}"
)
producer = Producer(
make_config(
"orders-producer",
**{
"acks": "all",
"enable.idempotence": True,
"linger.ms": 5,
},
)
)
try:
for order_id in ("order-1001", "order-1002", "order-1003"):
event = {"order_id": order_id, "status": "created"}
key = b"customer-42"
while True:
try:
producer.produce(
topic="orders",
key=key,
value=json.dumps(event).encode("utf-8"),
on_delivery=on_delivery,
)
break
except BufferError:
# Serve delivery callbacks while the local queue is full.
producer.poll(1)
producer.poll(0)
finally:
remaining = producer.flush(30)
if remaining:
raise TimeoutError(f"{remaining} messages were not delivered")
if failures:
raise RuntimeError(f"Delivery failed: {failures}")
def consume():
consumer = Consumer(
make_config(
"orders-consumer",
**{
"group.id": "orders-processors",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"isolation.level": "read_committed",
},
)
)
try:
consumer.subscribe(["orders"])
while True:
message = consumer.poll(1.0)
if message is None:
continue
if message.error():
if message.error().code() == KafkaError._PARTITION_EOF:
continue
raise KafkaException(message.error())
event = json.loads(message.value().decode("utf-8"))
print(f"Processing {event}")
# Commit only after the business side effect succeeds.
consumer.commit(message=message, asynchronous=False)
except KeyboardInterrupt:
pass
finally:
consumer.close()
if __name__ == "__main__":
parser = argparse.ArgumentParser()
parser.add_argument("mode", choices=("produce", "consume"))
args = parser.parse_args()
if args.mode == "produce":
produce()
else:
consume()
Run each role separately:
python kafka_app.py produce
python kafka_app.py consume
For a secured cluster, set KAFKA_SECURITY_PROTOCOL=SASL_SSL, provide the SASL environment variables, and add --command-config /path/to/client.properties to the administrative topic command.
Production Operations
Monitoring: Track consumer lag, under-replicated and offline partitions, ISR changes, request latency, disk usage, producer errors, and rebalance frequency.
Retention: Configure retention.ms or retention.bytes. Use cleanup.policy=compact for state-like topics that retain the latest value per key.
Disaster recovery: Replication is not a backup. Use cross-region replication or mirroring and regularly test restoration procedures.
Security and upgrades: Use TLS or mTLS, SASL, least-privilege ACLs, and secret-manager injection. Upgrade brokers in a supported rolling sequence before upgrading clients.
Best Practices and Common Pitfalls
Use acks=all and idempotence for retry-safe, durable production.
Treat processing and offset commit as separate steps; duplicates are preferable to data loss.
Route malformed or repeatedly failing records to a dead-letter topic.
Avoid excessive partition counts and use snappy or zstd compression where supported.
Do not commit credentials; inject the SASL password through the runtime secret store.
74
How to create Kafka topics and inspect consumer groups with CLI commands
KnownRevise
Use Apache Kafka CLI tools to create and describe topics with kafka-topics.sh, then inspect consumer-group membership, committed offsets, and lag with kafka-consumer-groups.sh. The commands connect through --bootstrap-server, so they do not require direct access to every broker.
Read full answerHide full answer
Applies toApache Kafka 4.x in KRaft mode
Code Example
Run these commands from the Kafka installation directory:
# Create a topic with six partitions and three replicas.
bin/kafka-topics.sh \
--bootstrap-server broker-1:9092 \
--create \
--if-not-exists \
--topic orders \
--partitions 6 \
--replication-factor 3 \
--config min.insync.replicas=2
# List topics.
bin/kafka-topics.sh \
--bootstrap-server broker-1:9092 \
--list
# Show partition leaders, replicas, in-sync replicas, and topic configuration.
bin/kafka-topics.sh \
--bootstrap-server broker-1:9092 \
--describe \
--topic orders
# List consumer groups known to the cluster.
bin/kafka-consumer-groups.sh \
--bootstrap-server broker-1:9092 \
--list
# Show members, partition assignments, committed offsets, and lag.
bin/kafka-consumer-groups.sh \
--bootstrap-server broker-1:9092 \
--describe \
--group order-processor
If the cluster requires TLS or SASL authentication, pass a client properties file to both tools:
Group state, active members, assignments, current offset, log-end offset, and lag
Consumer-group lag is approximately:
lag = log-end offset - current committed offset
It represents records not yet consumed by the group, although short-lived differences are normal while consumers process records or commits are delayed.
How It Works
Topic creation: kafka-topics.sh sends a create request through a broker to the active KRaft controller. The controller creates the partition metadata and assigns replica leaders.
Replication: Each partition has one leader and configured follower replicas. Only leaders serve reads and writes for that partition.
In-sync replicas: Replicas that have caught up appear in the ISR. With min.insync.replicas=2, an acks=all producer succeeds only while at least two replicas are in sync.
Consumer groups: Kafka’s group coordinator assigns partitions among group members. Committed offsets are stored in the internal __consumer_offsets topic; do not modify that topic directly.
Ordering: Kafka preserves record order within a partition, not across an entire topic. A consumer group normally assigns each partition to only one member at a time.
Guarantees and Caveats
CLI inspection does not change producer or consumer delivery semantics.
For reliable at-least-once processing, disable automatic offset commits and commit the offset only after successful processing. A crash before the commit can cause reprocessing and duplicates.
Kafka’s transactional read-process-write flow provides exactly-once behavior within Kafka. It requires a transactional producer with transactional.id, enable.idempotence=true, and acks=all; the consumer must use isolation.level=read_committed. Offsets and output records must be included in the same transaction.
Kafka Streams applications can use processing.guarantee=exactly_once, subject to correctly configured transaction-state replication.
Exactly-once does not make non-transactional databases, HTTP calls, or other external side effects atomic.
Production Practices
Use replication factor 3 across separate failure domains and set min.insync.replicas=2 for important topics. Creation fails if the cluster cannot satisfy the requested replication factor.
Monitor consumer lag, under-replicated partitions, offline partitions, request latency, disk utilization, and broker failures. Sustained lag is usually a capacity, processing-time, or rebalance problem.
Expect temporary command failures during controller elections, broker outages, coordinator transitions, or network partitions; retry with bounded backoff and alerts.
Grant least-privilege ACLs for topic creation, topic description, group description, and cluster metadata. Administrative operations should use encrypted connections and protected client configuration files.
75
How do you design an event-driven order-processing system with Kafka?
KnownRevise
An event-driven order-processing system uses Kafka as the durable event backbone: services publish immutable events, partition them by orderId, and consume them through groups with idempotent business processing. This provides at-least-once transport while avoiding duplicate business effects.
Controllers: Run three KRaft controllers across failure domains. Kafka 4.x does not use ZooKeeper.
Brokers and partitions: Each topic has partitions, each with one leader and replicas. Start with replication factor 3 and min.insync.replicas=2 across availability zones.
Producers: Set the record key to orderId, preserving ordering only for that order. Include eventId, schema version, event type, timestamp, and payload.
Consumer groups: Give each independently scalable service its own group, such as order-processors or notifications. A group assigns each partition to one member; committed offsets are stored in the internal __consumer_offsets topic by default.
Database bridge: Write the business row and an outbox row in one database transaction. Debezium on Kafka Connect can capture the outbox change into Kafka, avoiding an atomic database-to-Kafka commit.
Delivery Guarantees
Semantics
Configuration and behavior
Trade-off
At most once
Use producer acks=0, or commit a consumer offset before processing
Fastest, but events can be lost
At least once
Use acks=all, enable.idempotence=true, and commit offsets after processing
Recommended; retries or crashes can cause duplicates
Exactly once within Kafka
Use a transactional producer with a unique, stable transactional.id, include source offsets in the transaction, and consume with isolation.level=read_committed
Kafka read-process-write is atomic; external systems are not
Idempotent producers suppress retries caused by producer failures, but not intentional or cross-session duplicate events. Apply the inbox pattern: enforce a unique eventId and perform the inbox insert and order-state update in the same database transaction. This provides effectively-once business effects over at-least-once delivery.
Data Management
Retention: Use cleanup.policy=delete for event history. A seven-day retention period is illustrative; regulated records may require longer retention or archival.
Compaction: Use a separate compacted order-state topic keyed by orderId. Compaction is eventual; tombstones mark keys as deleted and remain for delete.retention.ms.
Schema Registry: Register Avro, JSON Schema, or Protobuf schemas in Schema Registry rather than embedding schemas in every event. Schema Registry persists schemas in its internal _schemas topic by default; use BACKWARD compatibility unless another migration policy is required.
Kafka Connect: Use Debezium for CDC and connector-specific sinks for projections. Source exactly-once support requires workers with exactly.once.source.support=enabled and a capable connector (exactly.once.support=required); sink support is connector-specific, so otherwise use idempotent sinks.
Kafka Streams: Use it for stateful aggregation within the event pipeline. Set processing.guarantee=exactly_once_v2 when its Kafka read-process-write transaction model is required.
Operations and Failure Modes
Monitor consumer lag, rebalance frequency, produce and consume latency, error rates, disk usage, under-minimum-ISR partitions, offline partitions, leader elections, and broker quotas.
With acks=all and min.insync.replicas=2, writes become unavailable if the in-sync replica count falls below two. A broker failure normally causes leader election and continued service when sufficient replicas remain.
Poison messages can block a partition. Move repeatedly failing events to bounded retry topics and then a dead-letter topic, preserving the original key and adding error metadata.
Keep processing within max.poll.interval.ms; reduce batch work or use bounded batches rather than merely increasing the timeout. Monitor hot orderId keys and partition traffic.
Use TLS for clients and inter-broker traffic, SASL/SCRAM or OAUTHBEARER authentication, least-privilege ACLs, and no unauthenticated production listener.
Replication is not a backup. Use MirrorMaker 2 for another region, archive important events to durable object storage, and test restoration procedures.
Code Example
Create event and compacted-state topics with the required ISR threshold:
The following consumer demonstrates inbox deduplication before committing its offset. In production, put the order-state mutation in the same database transaction as the inbox insert.
import json
import sqlite3
from contextlib import closing
from confluent_kafka import Consumer, KafkaError
DB = "orders.db"
BOOTSTRAP = "kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092"
with closing(sqlite3.connect(DB)) as db, db:
with closing(db.execute(
"CREATE TABLE IF NOT EXISTS processed_events "
"(event_id TEXT PRIMARY KEY NOT NULL, payload TEXT NOT NULL)"
)):
pass
def persist_once(raw):
event = json.loads(raw)
with closing(sqlite3.connect(DB)) as db, db:
with closing(db.execute(
"INSERT INTO processed_events(event_id, payload) VALUES (?, ?) "
"ON CONFLICT(event_id) DO NOTHING",
(event["eventId"], raw.decode("utf-8")),
)):
pass
consumer = Consumer({
"bootstrap.servers": BOOTSTRAP,
"group.id": "order-processors",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"isolation.level": "read_committed",
})
try:
consumer.subscribe(["order-events"])
while True:
message = consumer.poll(1.0)
if message is None:
continue
if message.error():
if message.error().code() == KafkaError._PARTITION_EOF:
continue
raise RuntimeError(str(message.error()))
persist_once(message.value())
consumer.commit(message=message, asynchronous=False)
except KeyboardInterrupt:
pass
finally:
consumer.close()
Production Best Practices
Test broker, network, controller, disk, consumer, rebalance, and replay scenarios.
Exercise schema upgrades and partition expansion before production rollout.
Keep sensitive payloads out of logs, encrypt stored data where required, and never reuse an eventId for different business actions.
76
How can Kafka guarantee exactly-once payment event processing?
KnownRevise
Exactly-once processing requires Kafka offsets, internal state, and produced records to be committed atomically. Kafka can provide this guarantee for Kafka-to-Kafka processing, but an external payment or database operation also requires an idempotency key or transactional-outbox pattern because Kafka cannot atomically commit an external system.
Read full answerHide full answer
Applies toApache Kafka 4.x with Kafka Streams and KRaft
Guarantee Boundary
Scenario
Guarantee
At-least-once consumer
The handler may run again after a crash, causing duplicate effects
Kafka transactional processing
State, offsets, and Kafka output commit atomically, producing one visible output per accepted input record
External payment or database
Not covered by Kafka transactions; the external operation must be idempotent
Exactly-once also does not mean application code executes only once. A Kafka Streams processor can rerun after an aborted transaction or crash. The committed Kafka result is exactly once, while the business operation must tolerate retries safely.
Required Architecture
Use Kafka Streams with processing.guarantee=exactly_once_v2, or manually coordinate a transactional producer with consumer offsets.
Enable producer idempotence with enable.idempotence=true, acks=all, and no more than five in-flight requests per connection.
Use a stable transactional.id for manually transactional producers. Kafka Streams manages transactional IDs using its application.id.
Commit the consumed offset, state-store change, and output record in the same transaction.
Configure downstream consumers with isolation.level=read_committed; otherwise they can observe records from aborted transactions.
Use a stable payment_id as the partition key so all events for one payment stay in order, and carry a unique event_id in each record for deduplication. Ordering holds per partition, not globally across the topic.
Ensure each payment event ID is unique upstream. Idempotent production prevents retry duplicates, while topic compaction only controls long-term log size and does not provide immediate deduplication.
How It Works
With exactly_once_v2, Kafka Streams runs one transaction per stream thread, covering all of that thread’s tasks. It writes output records and updated state to internal changelog topics, then commits those records together with the consumed source offsets. If the process fails before commit, Kafka aborts the transaction and reprocesses the input after restoration.
Consumers using read_committed see only committed transactional records. Ordering is guaranteed within each partition; Kafka does not guarantee global ordering across partitions.
For an external payment provider, retry the same operation with the event_id as the provider’s idempotency key. For a database, enforce a unique constraint on event_id and update the payment record and an outbox row in one database transaction. A distributed transaction between Kafka, a database, and a payment provider generally does not exist.
Code Example
Create replicated input and output topics across three brokers. Event topics use delete; compacting them by key would discard earlier events of a payment:
for topic in payments payments-processed; do
bin/kafka-topics.sh \
--bootstrap-server kafka-1.example.com:9092,kafka-2.example.com:9092,kafka-3.example.com:9092 \
--create \
--topic "$topic" \
--partitions 12 \
--replication-factor 3 \
--config cleanup.policy=delete
done
Replace the deterministic mapValues operation with the payment business logic. If that logic calls a provider or database, implement the separate idempotency mechanism described above.
A transaction timeout aborts work and causes reprocessing. Keep business operations fast and configure timeouts consistently across producer, broker, and application.
Kafka Streams state is restored from changelog topics. Test restoration, configure rack awareness, and use standby replicas where availability requirements justify them.
Perform rolling upgrades only with compatible clients and verify that transactional processing remains enabled.
Protect brokers with TLS and SASL, and restrict topic, consumer-group, and transactional-topic ACLs.
Best Practices and Pitfalls
Treat Kafka exactly-once as an end-to-end design, not a single broker setting.
Kafka Connect provides exactly-once only for connectors that explicitly support transactional source offsets and sinks. Otherwise, use at-least-once plus a unique external key.
Schema Registry can enforce payment-event schemas and compatibility, but it does not provide exactly-once delivery by itself.
Do not assume global event ordering; partition by a stable key such as payment_id.
Keep the payment event ID stable across every retry, state update, outbox record, and external API call.
77
How do you troubleshoot rapidly growing Kafka consumer group lag?
KnownRevise
Consumer lag is the difference between a partition’s log-end offset and the offset committed by its consumer group. Rapidly increasing lag usually means production exceeds the group’s effective processing capacity, although stopped consumers, rebalances, broker failures, or downstream outages can cause the same symptom.
Read full answerHide full answer
Applies toApache Kafka 3.6-4.x
Diagnose the Bottleneck
Kafka stores each topic in partitions whose logs are replicated across brokers; one partition is assigned to only one member of a consumer group.
Committed versus live position: --describe reports the committed offset. With auto-commit enabled, it can overstate the actual backlog by up to the commit interval; correlate it with consumer metrics such as records-lag-max and the processed-record rate.
Active members and assignment: Check group state, member count, and lag per partition. A group with no active consumers cannot make progress, and a hot partition limits parallelism within that group.
Rate comparison: Compare producer and consumer rates; a sustained difference explains continued growth.
Rebalance and downstream health: Frequent rebalances interrupt polling; database calls, HTTP requests, locks, CPU saturation, or a downstream outage can make consumers appear healthy while processing very slowly.
Broker and metadata health: Check unavailable or under-replicated partitions, disk, network, request latency, and send latency. Kafka 3.6 supports KRaft or ZooKeeper; Kafka 4.x is KRaft-only, so also inspect controller-quorum health where applicable.
Other client types: For Kafka Connect, inspect connector and task status plus task lag; for Kafka Streams, inspect thread/client lag, processing and restore latency, and state-store performance; for Schema Registry, inspect compatibility or validation errors that cause records to be rejected or retried as poison records.
Offset lag is an offset delta, not necessarily the number of retained records: compaction and control or aborted transaction records can leave offset gaps. Estimate backlog age from record timestamps and use it alongside lag.
Safely Reduce the Backlog
Scale consumers when partitions and downstream capacity allow it. Add members up to the number of partitions assigned to the topic; additional consumers remain idle.
Fix hot partitions if concurrency is limited. Increasing partition count enables parallelism, but it can change the key-to-partition mapping, so validate ordering requirements first.
Reduce processing time by batching downstream writes, removing unnecessary remote calls, caching stable lookups, or using asynchronous processing with controlled concurrency.
Tune polling and fetching carefully. Increasing max.poll.records reduces poll overhead but increases batch size and the chance of exceeding max.poll.interval.ms.
Reduce rebalance impact with CooperativeStickyAssignor and, where appropriate, static membership through a unique, stable group.instance.id.
Control upstream traffic during an incident if necessary and with business approval. This slows backlog growth and buys drain time, but it does not extend retention or prevent existing records from expiring.
Do not blindly increase max.poll.interval.ms; it allows slow processing to continue but does not improve throughput and can delay failure detection.
Preserve Delivery and Ordering Guarantees
Kafka provides ordering within a partition, not across partitions. Consumer-group membership does not change that boundary.
At-least-once: Disable automatic offset commits, process the records successfully, and then commit their offsets. A crash before the commit causes reprocessing but avoids data loss.
Exactly-once: For a consume-transform-produce workflow, use an idempotent transactional producer with a stable, unique transactional.id per producer instance, send consumed offsets to the same transaction with sendOffsetsToTransaction, and consume with isolation.level=read_committed.
Read-committed consumers: They stop at the last stable offset when a producer transaction remains open. An unusually long-running transaction can therefore look like lag.
Retries: Bound retries and isolate poison messages. Infinite retries inside the poll loop can trigger consumer eviction.
Exactly-once processing adds transaction-coordination overhead and does not make inherently slow application logic faster. External database or HTTP side effects still require idempotency or an outbox.
Code Example
Inspect the group, partition layout, and broker replication state:
For a static consumer, also configure a unique identifier:
group.instance.id=order-processor-1
Production Practices and Pitfalls
Monitor partition lag, backlog age, processing rate, rebalance frequency, commit failures, broker request latency, under-replication, disk usage, and producer backpressure.
Alert on trends rather than a single offset threshold; a growing backlog is usually more actionable than a temporarily high value.
Check delete-retention deadlines and disk headroom. Monitoring alone cannot extend retention, so use an approved retention change or recovery plan before data expiry.
Use replication and tested cluster-level backups; consumer offsets are not a substitute for data backups.
Prefer TLS and SASL authentication, and grant only required topic and group ACL permissions.
Before rolling upgrades, confirm compatibility, healthy replication, sufficient capacity, and a plan for draining or reassigning consumer groups.
Avoid resetting offsets to the latest during an incident unless data loss is explicitly accepted; that hides the backlog rather than resolving it.
78
How do you migrate relational database polling to Kafka events?
KnownRevise
Use a transactional outbox to convert committed relational changes into Kafka events without an unsafe database-to-Kafka dual write. Migrate with parallel shadow validation, then switch consumers and retire polling only after parity, reliability, and rollback criteria are met.
Write the business change and an event_outbox row in the same database transaction.
In a bounded relay transaction, select unpublished rows with FOR UPDATE SKIP LOCKED, publish them to Kafka, wait for acknowledgement, set published_at, and commit the transaction. Roll back on failure so another relay can retry.
Give each event an immutable event_id, schema_version, event type, and aggregate key. Use the aggregate ID as the Kafka key to preserve ordering for that entity.
Partition a topic across multiple brokers. For example, use 12 partitions, replication factor 3, and min.insync.replicas=2.
Kafka 4.x uses KRaft: controllers maintain cluster metadata, while brokers store and replicate partition data.
For high-volume or low-latency change capture, a Kafka Connect source such as Debezium can replace the custom relay when every captured change is mapped into the required event contract.
Use Kafka Connect for sink connectors, Kafka Streams for stateful transformations, and Schema Registry for versioned Avro or JSON Schema payloads.
Cutover Process
Record the current poller’s correctness, freshness, throughput, and recovery requirements.
Create the Kafka topic, ACLs, schema subjects, and consumer groups.
Deploy the outbox and relay while continuing to run the existing polling workflow.
Run shadow consumers against Kafka, compare their results with the polling path, and suppress duplicate production side effects.
Switch production consumers to Kafka, monitor lag and discrepancies, and stop the poller after the rollback window. Retain the outbox temporarily for recovery and audit.
Delivery and Ordering Guarantees
Kafka preserves ordering within a partition, not across the topic. Using one aggregate ID as the key maps that aggregate to one partition.
Enable producer idempotence with acks=all, enable.idempotence=true, and max.in.flight.requests.per.connection=5.
Idempotence suppresses duplicates caused by retrying the same producer sequence. It does not prevent a second application-level publish, such as an outbox relay retrying after Kafka acknowledged the event but before its database transaction committed. Consumers must deduplicate using event_id, commonly with a database inbox table and unique constraint.
For at-least-once consumption, set enable.auto.commit=false and commit the next partition offset only after processing the event and committing its database side effect. A crash between those commits causes a duplicate rather than a loss.
Kafka exactly-once semantics require transactional producers and consumers using isolation.level=read_committed, with consumed offsets included in the producer transaction. They do not automatically make an arbitrary external database write exactly once; use inbox/outbox coordination or an equivalent mechanism.
Code Example
Create the outbox table:
CREATE TABLE event_outbox (
event_id uuid PRIMARY KEY,
schema_version smallint NOT NULL,
topic text NOT NULL,
event_key text NOT NULL,
event_type text NOT NULL,
payload jsonb NOT NULL,
created_at timestamptz NOT NULL DEFAULT now(),
published_at timestamptz
);
CREATE INDEX event_outbox_unpublished_idx
ON event_outbox (created_at)
WHERE published_at IS NULL;
Create the topic with client properties supplied through --command-config:
import json
import os
from confluent_kafka import Producer
producer = Producer({
"bootstrap.servers": "kafka.example.com:9093",
"security.protocol": "SASL_SSL",
"sasl.mechanism": "SCRAM-SHA-512",
"sasl.username": "outbox-relay",
"sasl.password": os.environ["KAFKA_PASSWORD"],
"enable.idempotence": True,
"acks": "all",
"max.in.flight.requests.per.connection": 5,
"delivery.timeout.ms": 120000,
})
def publish_outbox_row(row):
errors = []
def delivered(err, message):
if err is not None:
errors.append(str(err))
value = json.dumps({
"event_id": str(row["event_id"]),
"schema_version": row["schema_version"],
"event_type": row["event_type"],
"payload": row["payload"],
}).encode("utf-8")
producer.produce(
row["topic"],
key=row["event_key"].encode("utf-8"),
value=value,
headers=[("event_type", row["event_type"])],
on_delivery=delivered,
)
if producer.flush(130) != 0 or errors:
raise RuntimeError("Kafka delivery did not complete")
publish_outbox_row({
"event_id": "2fd33344-45d3-4c26-b5cf-8e724fe73122",
"schema_version": 1,
"topic": "orders.v1",
"event_key": "order-42",
"event_type": "OrderPlaced",
"payload": {"order_id": "order-42", "status": "paid"},
})
Production Operations
Monitor consumer lag and lag age, under-replicated and offline partitions, broker disk usage, produce failures, retry volume, and dead-letter queue growth.
Configure bounded consumer retries and send repeatedly failing events to a versioned DLQ topic.
Use cleanup.policy=delete with an appropriate retention.ms for immutable event history. Use log compaction only for latest-state or changelog topics.
Enforce TLS and SASL authentication, least-privilege topic and group ACLs, Schema Registry authorization, and secrets management. Provision KAFKA_PASSWORD at runtime from a secret manager; never commit credentials.
Replication is not a backup. Use MirrorMaker 2 or another tested cross-region replication design, define recovery objectives, and rehearse restoration.
Upgrade brokers in a rolling manner after checking compatibility with Kafka Connect, Schema Registry, and clients. Use FORWARD or FULL compatibility when consumers may upgrade after producers; BACKWARD assumes consumers upgrade first.
79
How do you design a real-time analytics pipeline with Kafka?
KnownRevise
A real-time analytics pipeline should use Kafka as a durable event backbone, decoupling ingestion, stream processing, and analytical sinks. Event-driven architecture provides replay, horizontal scalability, and controlled delivery semantics when topics, partitions, replication, and consumer offsets are designed explicitly.
Source producers publish events to an append-only, versioned raw topic such as orders.v1.
Partitions distribute events across brokers. Use a stable key such as customer_id when events for one entity must remain ordered.
Brokers store partition leaders and followers. With replication factor 3, a leader failure can be replaced without losing acknowledged data.
Consumer groups divide partitions among consumers. Each group maintains its own offsets in __consumer_offsets.
Stream processors validate, enrich, aggregate, and republish events. Analytical sinks such as ClickHouse, Elasticsearch, or a data warehouse receive processed results through Kafka Connect or application-specific consumers.
In KRaft mode, deploy an odd-sized controller quorum, normally three controllers across failure domains, plus multiple brokers. Controllers manage metadata; brokers handle the data path.
Topic and Retention Design
Set partitions according to expected throughput and required consumer parallelism; avoid excessive partitions and key-induced hotspots.
Use replication factor 3 and min.insync.replicas=2 for production durability.
Retain raw events long enough to replay and rebuild downstream state.
Use cleanup.policy=delete for event streams and cleanup.policy=compact for keyed state topics. Emit tombstones when compact keys must be removed.
Place replicas across availability zones within a cluster. Cross-region disaster recovery requires MirrorMaker 2 or another replication system; replication alone is not a backup.
Delivery Guarantees
At least once: Disable auto-commit, process the event, then commit its offset. A crash after processing but before committing causes redelivery, so the sink must deduplicate using an event ID or equivalent.
Exactly once for Kafka read-process-write: Use a transactional producer with a transactional.id that is stable across restarts and unique among concurrently active producers, enable idempotence, and configure consumers with isolation.level=read_committed. Atomically include output records and consumed offsets in the transaction using send_offsets_to_transaction.
Ordering: Kafka preserves ordering within a partition, not across a topic. Route related events to the same partition. Idempotent production permits up to five in-flight requests per connection while preserving order.
Exactly-once processing does not automatically include writes to an external database. External effects still require idempotency or a coordinated transaction.
Streaming Ecosystem
Kafka Streams provides stateful operators, windowed aggregation, and repartitioning. State changes are backed by changelog topics for recovery.
Kafka Connect runs source and sink connectors with independent scaling and offset management. Connector-specific transactional support determines whether exactly-once behavior is possible.
Schema Registry stores Avro, JSON Schema, or Protobuf schemas. Enforce compatibility such as BACKWARD to prevent incompatible event evolution.
The following producer is idempotent, while the consumer implements manual offset commits:
import os
from confluent_kafka import Producer, Consumer
bootstrap = os.getenv("KAFKA_BOOTSTRAP", "kafka.example.com:9092")
security = {
"bootstrap.servers": bootstrap,
"security.protocol": "SASL_SSL",
"sasl.mechanism": "SCRAM-SHA-512",
"sasl.username": os.environ["KAFKA_USERNAME"],
"sasl.password": os.environ["KAFKA_PASSWORD"],
}
producer = Producer({
**security,
"enable.idempotence": True,
"acks": "all",
"max.in.flight.requests.per.connection": 5,
})
delivery_errors = []
def on_delivery(error, message):
if error is not None:
delivery_errors.append(str(error))
producer.produce(
"orders.v1",
key=b"customer-42",
value=b'{"event_id":"event-1","amount":125.50}',
on_delivery=on_delivery,
)
producer.flush()
if delivery_errors:
raise RuntimeError("; ".join(delivery_errors))
consumer = Consumer({
**security,
"group.id": "analytics",
"enable.auto.commit": False,
"auto.offset.reset": "earliest",
"isolation.level": "read_committed",
})
consumer.subscribe(["orders.v1"])
try:
while True:
message = consumer.poll(1.0)
if message is None:
continue
if message.error():
print(f"Kafka error: {message.error()}")
continue
# Replace this with an idempotent, durable sink write.
print(message.value().decode("utf-8"))
consumer.commit(message=message, asynchronous=False)
except KeyboardInterrupt:
pass
finally:
consumer.close()
Production Operations
Monitor consumer lag by partition and lag age, offline or under-replicated partitions, broker disk usage, request latency, JVM garbage collection, and producer error rates.
Route repeatedly failing records to a dead-letter topic; otherwise one poison event can block a partition.
Use TLS for encryption in transit, SASL_SSL authentication, and least-privilege ACLs for topics, consumer groups, and Schema Registry.
Roll broker upgrades one at a time, checking ISR and replication health after each step. Do not lose controller quorum during the upgrade.
Test restoration using a separate region or MirrorMaker 2. Rebuild downstream state from retained raw events, back up schema and configuration metadata, and periodically validate backup integrity.
80
How should you choose a retention period for an Apache Kafka topic?
KnownRevise
Choose the topic’s retention period from the replay window the business requires, plus enough time for consumers to recover from outages, while remaining within the cluster’s storage capacity. The policy must match whether the topic contains a complete event history, current state per key, or both.
Read full answerHide full answer
Applies toApache Kafka 4.x
How Kafka Retention Works
For cleanup.policy=delete, Kafka removes whole log segments rather than individual records. The active segment is not deleted, so actual data lifetime can exceed the configured period until a segment rolls.
retention.ms: Maximum record age. The default is seven days; -1 retains data indefinitely.
retention.bytes: Maximum log size per partition. The default is -1, meaning no size limit.
Segment rolling: segment.ms and segment.bytes determine when a new segment starts. Smaller segments provide more precise cleanup but create more files.
Deletion timing: Retention checks and filesystem deletion are periodic rather than instantaneous. The broker-level file.delete.delay.ms controls the delay between physical file deletions.
Combined constraints: With cleanup.policy=delete, age or size limits can cause deletion. retention.bytes can therefore remove old segments before the nominal time window, so it is a capacity safeguard rather than a precise retention guarantee.
Choosing a Defensible Period
Base the decision on concrete recovery requirements:
Replay requirement: Determine how far back an application must replay events. For example, a fraud pipeline might require 30 days for investigations.
Consumer outage tolerance: Include the longest acceptable outage plus catch-up time. A consumer that falls behind by seven days may find that the required records have already expired.
Recovery objective: Align the window with the business’s acceptable recovery point.
Storage capacity: Estimate retained bytes as aggregate ingress bytes/second × retention seconds × replication factor, then add headroom for bursts and segment overhead. Remember that retention.bytes applies independently to each partition.
Compliance: Longer retention increases cost and the amount of sensitive data retained. Apply data minimization, encryption, ACLs, and auditing.
Consumer-group offsets are retained separately through offsets.retention.minutes. This expiry begins after a consumer group becomes empty; deleting event data does not delete the corresponding offsets.
Delete vs. Compact
Policy
Behavior
Suitable use
Important caveat
delete
Removes old segments based on age or size
Logs, immutable events, replayable streams
Data may disappear before the exact retention.ms value
compact
Retains the latest surviving record for each key
State stores and CDC-style topics
Older records are removed only after compaction
compact,delete
Combines compaction with segment expiry
Bounded state topics
Both compaction and expiry behavior must be tuned
For a compacted topic, a null value is a tombstone that deletes the key during compaction. delete.retention.ms controls how long that tombstone is retained after compaction to prevent out-of-order processing from resurrecting the key. Compaction retains surviving records in partition order, but records for the same key must consistently use the same partition.
Operational and Security Considerations
Consumer lag: Track lag against the oldest available record, not merely the latest offset. Data can expire while a consumer still has an earlier committed offset.
Delivery guarantees: Ordering is guaranteed only within a partition. For at-least-once consumption, disable auto-commit and commit offsets only after processing; a crash can still repeat an external side effect. Kafka 4.x producers enable idempotence by default, but that does not make consumer side effects exactly-once.
Exactly-once processing: A Kafka read-process-write workflow requires idempotent production, a stable and unique transactional.id, transactional offset commits, enable.auto.commit=false, and consumers using isolation.level=read_committed. Kafka Streams applications use processing.guarantee=exactly_once_v2. External systems still require transactional integration or an outbox pattern.
Replication: Segment deletion is propagated across replicas asynchronously. Monitor under-replicated and offline partitions.
Storage: Failed cleanup can fill disks and destabilize brokers. Alert on filesystem utilization, partition growth, consumer lag, and the age of the oldest available record.
Backups: Kafka retention is not archival. Mirror important topics to durable external storage when longer-term recovery is required.
Security: Restrict topic-configuration changes to authorized administrators and use topic ACLs to control producer and consumer access.
Changes: Topic retention settings are dynamic and normally do not require a broker restart, but existing and active segments affect when data disappears.
Code Example
Set a seven-day time limit and a 10 GiB per-partition safety threshold: