Apache Kafka End to End: Cluster, Partitions, Brokers & Message Flow Across Partitions

Apache Kafka is a distributed commit log: producers write events into topics, brokers store them on disk in partitions, and consumers read at their own pace. For NestJS SaaS and microservices, that means order events, billing webhooks, and audit streams can fan out without coupling every service to a shared database write path.

This guide walks the end-to-end lifecycle — cluster membership, how partitions hold ordered logs, how brokers accept and replicate messages, and how consumer groups share work across partitions — plus how Cyber Infoware kits fit event-driven backends (RabbitMQ today, Kafka when your throughput and retention needs grow).

The Kafka cluster: brokers, controllers, and topics

• Broker — a Kafka server process that stores partition logs and serves produce/fetch RPCs
• Cluster — multiple brokers that share topic metadata and replicate partitions for durability
• Controller — elected broker that manages partition leadership and membership changes
• Topic — a named stream (e.g. `orders.created`) split into one or more partitions

A minimal production sketch: 3 brokers, replication factor 3, `min.insync.replicas=2`, topics sized by expected parallelism — not by guesswork on day one.

Partitions: the unit of ordering and scale

1. Each partition is an append-only log with monotonically increasing offsets (0, 1, 2, …)
2. Ordering is guaranteed inside one partition, not across the whole topic
3. More partitions → more parallel consumers in a group (up to one active consumer per partition)
4. Partition count is hard to shrink later — start with headroom, not infinite shards

How messages land on a partition

• With a key (e.g. `tenant_id` or `order_id`) — hash(key) % partition_count picks the partition; same key stays ordered
• Without a key — sticky/round-robin batching across partitions; no per-entity order
• Explicit partition — rare; usually reserved for tooling or migrations

If you need "all events for order X in order," put `order_id` in the key. If you need "global total order for every event," you need one partition — and you just limited horizontal scale.

Broker lifecycle: accept, replicate, serve

1. Produce — client sends a batch to the partition leader broker
2. Append — leader writes to its local log (and page cache); durability depends on `acks` (`1` vs `all`)
3. Replicate — follower brokers fetch from the leader; in-sync replicas (ISR) track who is caught up
4. Acknowledge — when ISR policy is satisfied, the producer gets success
5. Fetch — consumers pull from leaders (or preferred replicas) starting at their committed offset
6. Retain — logs roll by size/time; compacted topics keep the latest value per key

Under load, brokers are disk- and network-bound. Large batches, compression (`lz4`/`zstd`), and enough partitions to avoid hot keys keep "server loading messages" healthy — a single hot partition (one giant tenant) will not scale just because you added brokers.

Across partitions: consumer groups and rebalancing

• A consumer group shares a topic: each partition is assigned to at most one member in the group
• 3 partitions + 3 consumers ≈ one partition each; 4 consumers means one sits idle until rebalance
• On join/leave, the group rebalances — assignments move; design handlers to be idempotent
• Offsets are committed per partition — progress on P0 does not imply progress on P1
• Different groups reading the same topic each get a full copy of the stream (fan-out)

End-to-end example: `orders.events`

• Topic `orders.events` with 6 partitions, RF=3
• NestJS Order Service produces with key=`orderId` so status updates stay ordered per order
• Billing consumer group (2 pods) processes payment side effects across partitions
• Audit consumer group (1 pod) writes an immutable trail — independent lag from billing
• If billing lags on partition 2 only, other partitions keep draining — lag is per partition

Production checklist

• Choose keys for entity ordering; monitor partition skew (hot keys)
• Use `acks=all` + adequate ISR for money and identity events
• Idempotent producers / transactional outbox from NestJS when dual-write to DB + Kafka
• Consumer idempotency — rebalances and retries will redeliver
• Alert on under-replicated partitions, ISR shrink, and consumer lag by partition
• Capacity-plan disk retention separately from peak produce rate

Kafka vs queues you already know

• RabbitMQ / BullMQ — excellent for work queues, retries, and task fan-out with simpler ops
• Kafka — durable multi-consumer logs, replay, high throughput, partition-scaled parallelism
• Many SaaS teams start with RabbitMQ in the kit, then add Kafka for analytics CDC, high-volume domain events, or event sourcing style streams

What "done" looks like

Producers write keyed events, leaders replicate to ISR, consumer groups share partitions fairly, lag is visible per partition, and a redeploy does not lose the stream — only pauses consumption. Explore Cyber Infoware products or contact us for event-driven NestJS architecture and streaming design.