Kafka Deep Dive

Producers accumulate messages into batches per partition before sending:

Ack modes:

Production recommendation: acks=all + min.insync.replicas=2 + replication.factor=3. Write is only ACK’d when leader + ≥1 follower have the data.

Partition 0 on disk:
  00000000000000000000.log       ← segment (offsets 0-999)
  00000000000000001000.log       ← segment (offsets 1000-1999)
  00000000000000002000.log       ← active segment
  00000000000000000000.index     ← sparse offset → byte position
  00000000000000000000.timeindex ← timestamp → offset

The .index file is sparse — Kafka binary-searches it, then scans forward in the .log. This gives O(1) offset lookups despite the append-only structure.

One partition = one consumer per group. You lose all parallelism. Throughput is capped at what a single consumer can handle.

In practice, you design your partition key so ordering matters within a key (per-user, per-order), not globally. If you truly need global ordering (rare), consider whether you can relax the requirement to “causal ordering” and use techniques like logical clocks instead.

Producers: Broker 1 is still alive but can’t replicate to Brokers 2 and 3 — ISR shrinks to just [Broker 1]. If min.insync.replicas=2, writes fail with NotEnoughReplicasException because only 1 replica is in-sync. The producer gets errors and must retry or buffer.

Meanwhile: The controller (on the KRaft quorum, which Brokers 2 and 3 can still reach) detects Broker 1 is unreachable and elects a new leader from ISR members it can see. But if Broker 1 was the only ISR member reachable to itself, it’s a split-brain scenario.

Consumers: Consumer reads are also affected — they can only read up to the high watermark, which can’t advance without ISR replication.

Key insight: acks=all + min.insync.replicas=2 is a CP configuration (in CAP terms) — it chooses consistency (no data loss) over availability (partition becomes unavailable when quorum can’t be met).

The message is redelivered (at-least-once). The DB write happens again, potentially creating a duplicate row.

Fixes:

1. Idempotent consumer: Use a unique message ID (or topic+partition+offset) as a deduplication key in Postgres. Insert with ON CONFLICT DO NOTHING or check before writing.
2. Outbox pattern: Write the result + the offset into the same Postgres transaction. A separate process reads committed results and publishes downstream. The offset in Postgres becomes the source of truth, not __consumer_offsets.
3. Kafka transactions don’t help here because the Postgres write is outside Kafka’s transactional boundary. Exactly-once only applies to Kafka-to-Kafka pipelines.

Adding more consumers (>12): Useless. With 12 partitions, only 12 consumers can be active. Extra consumers sit idle — they’re just standby for failover.

Adding more partitions (e.g., 24): Increases parallelism (now 24 consumers can be active). But:

• hash(key) % N changes — existing keys remap to different partitions. In-flight data loses ordering guarantees during the transition.
• More partitions = more file handles, longer leader elections, higher controller memory.
• Partition count cannot be decreased. This is a one-way door.

Better alternatives before adding partitions:

• Optimize consumer processing (batch DB writes, async I/O)
• Use linger.ms and batch.size on the producer to reduce per-message overhead
• Check if consumers are I/O-bound rather than Kafka-bound

The new consumer starts reading from offset 0 and must process all 500 million messages before it catches up to real-time. This could take hours or days depending on processing speed.

Strategies:

1. Compacted topics: If the topic uses cleanup.policy=compact, Kafka retains only the latest value per key. The new consumer reads a smaller dataset (one message per key, not full history).
2. Snapshot + stream: Materialize the current state into a snapshot (e.g., a database dump or compacted topic), have the new service bootstrap from the snapshot, then start consuming from a recent offset.
3. Parallel bootstrap: Temporarily scale the new consumer group to many instances to chew through the backlog faster, then scale down.
4. Retention policy: If the topic has retention.ms or retention.bytes configured, old messages are already deleted. The consumer only processes what’s retained.

The real question: Should an event-sourced topic retain 2 years of raw events? Usually yes for audit, but the consumer doesn’t need to replay all of them — separate the “audit log” retention from the “bootstrap” mechanism.

The transactional.id bridges this gap. When a producer starts with a configured transactional.id, it calls initTransactions(). Kafka’s transaction coordinator maps the transactional.id to an internal ProducerID + epoch. If the producer crashes and restarts:

1. The new producer instance calls initTransactions() with the same transactional.id
2. The coordinator increments the epoch (fencing the old producer)
3. Any incomplete transactions from the old epoch are aborted
4. The new producer gets a fresh ProducerID + new epoch but the transactional.id provides continuity

Without transactional.id: A restarted producer gets a brand new ProducerID, and the broker can’t correlate it with the old one — sequence number deduplication only works within a single ProducerID’s lifetime. This is why the idempotent producer alone gives at-least-once (not exactly-once) across restarts. Transactions + transactional.id are required for true exactly-once.

No. Consumer groups are completely independent. Each group maintains its own offset per partition in __consumer_offsets. Consumer A and Consumer B read from the same partition leader, but:

• They have separate offset tracking
• They poll at their own pace
• A slow consumer in one group cannot back-pressure a consumer in another group
• They can even be at completely different offsets (A at offset 100, B at offset 50,000)

This is Kafka’s key advantage over traditional message queues (where a message is consumed by one consumer and deleted). The persistent log allows multiple consumer groups to read independently at different speeds — which is exactly why it works for the GPS scenario in the opening: pricing, ETA, data lake, and audit all read the same data independently.