Adam - Understanding Apache Kafka - Topics, Partitions, and Why the Log Is Everything

TL;DR

Kafka is a distributed, append-only commit log, not a message queue you drain.
A topic is split into partitions; each partition is an ordered, immutable sequence of records identified by an offset.
Consumer groups give you horizontal scaling: one partition is read by exactly one consumer in a group.
Ordering is guaranteed within a partition, never across a topic.
Replication + ISR + acks=all is how you avoid losing data when a broker dies.

Why I wanted to learn it

Kafka was the piece of my stack I'd heard about for years but never actually understood. I could say "event streaming" in a sentence; I couldn't explain what it really meant. What finally pushed me to learn it was my own code getting tangled: creating a task had to update search, fire off a notification, and bump some analytics - all inside the same request. If any one of those was slow, the whole thing felt slow. Every article I read said the same thing: decouple this with events. So I stopped nodding along and sat down to figure out how it actually works.

The mental model that unlocked it: it's a log

I kept trying to think of Kafka as a queue - messages go in, consumers take them out, they're gone. That's wrong and it held me back.

Kafka is an append-only log. Producers append records to the end. Records are not removed when read - they stay until a retention policy (time or size) ages them out. Consumers just track a pointer (an offset) into the log. This single design choice explains almost everything:

Multiple independent consumers can read the same data at their own pace.
A consumer can "rewind" by resetting its offset and reprocess history.
A new service can be added later and replay everything from the beginning.

The log is the source of truth. Once that clicked, the rest fell into place.

Topics and partitions

A topic is a named stream of records (e.g. task.created). For scale and parallelism, a topic is divided into partitions:

topic: task.created
  partition 0:  [0][1][2][3][4] →  (offsets)
  partition 1:  [0][1][2][3] →
  partition 2:  [0][1][2][3][4][5] →

Each partition is an ordered, immutable sequence. The offset is the record's position in its partition. Crucially:

Order is guaranteed within a partition, not across the topic. If two events must be processed in order, they must land in the same partition.
Which partition a record goes to is decided by its key. Same key → same partition. So I key task events by project_id to keep all events for one project ordered.

Consumer groups - the scaling trick

This is the part I found genuinely elegant. A consumer group is a set of consumers cooperating to read a topic. Kafka assigns each partition to exactly one consumer in the group:

3 partitions, 3 consumers → each reads one partition. Perfect parallelism.
3 partitions, 1 consumer → it reads all three.
3 partitions, 5 consumers → two consumers sit idle (partition count caps parallelism).

If a consumer crashes, Kafka rebalances and reassigns its partitions to the survivors. Meanwhile, a different group (say, the analytics service) reads the same partitions completely independently with its own offsets. One stream, many independent readers.

Brokers, replication, and not losing data

A Kafka cluster is made of brokers (servers). Partitions are spread across brokers, and each partition is replicated for fault tolerance:

One replica is the leader (handles all reads/writes); the others are followers that copy the leader.
The set of replicas currently caught up is the ISR (in-sync replicas).
If the leader broker dies, a follower from the ISR is promoted. No data loss - if you configured durability correctly.

That "if" is the producer acks setting, which I had to reason about carefully:

acks=0 - fire and forget. Fast, can lose data.
acks=1 - leader confirms. Lost if the leader dies before followers copy.
acks=all - leader waits for the full ISR. Slowest, safest.

For anything I care about, it's acks=all plus min.insync.replicas=2 so a write is only accepted if at least two replicas have it.

A producer, concretely

await producer.send({
  topic: "task.created",
  messages: [{
    key: String(task.projectId),       // same project → same partition → ordered
    value: JSON.stringify(task),
  }],
  acks: -1,                            // acks=all
});

And a consumer in a group:

const consumer = kafka.consumer({ groupId: "search-indexer" });
await consumer.subscribe({ topic: "task.created", fromBeginning: false });
await consumer.run({
  eachMessage: async ({ partition, message }) => {
    const task = JSON.parse(message.value!.toString());
    await searchIndex.upsert(task);
    // offset commits after success → at-least-once delivery
  },
});

Delivery semantics - the part that confused me longest

At-most-once: commit offset before processing. Fast, can drop on crash.
At-least-once: commit after processing. The common default - but a crash mid-commit means you reprocess, so handlers must be idempotent.
Exactly-once: Kafka's idempotent producer + transactions can give end-to-end exactly-once within Kafka. Across external systems, you still design idempotent consumers. I default to at-least-once + idempotency because it's simpler and robust.

What changed in my app

task.created now publishes one event. The search indexer, the notifier, and analytics each consume it in their own group, independently, at their own speed. The original API request returns immediately. One slow consumer no longer slows the user. That decoupling is the entire point.

Takeaways

Kafka is a log, not a queue. Internalize that and everything else follows.
Partitions are the unit of ordering and parallelism - choose your key deliberately.
Durability is a config decision: acks=all + min.insync.replicas.
Assume redelivery. Make consumers idempotent.

Now that work happens across multiple async services, a single failure is much harder to see. I can't debug what I can't observe - so I'm going to learn observability: logs, metrics, and distributed tracing.