Code Matrix

Apache Kafka Architecture Overview

Kafka follows a publish-subscribe model where producers send messages to Kafka topics, and consumers read from these topics. Apache Kafka is a distributed event streaming platform widely used for building real-time data pipelines and event-driven architectures. It provides high-throughput, fault tolerance, and scalability, making it a go-to choice for microservices communication, logging, analytics, and more.

Producers: Services that create and send events.

Consumers: Services that read and process events.

Topics: A logical bucket or category for events.

Broker: The actual servers that store and manage the data. Unlike traditional message brokers, Kafka persists data to disk, allowing it to be re-read (which is not the case with Rabbit MQ, Active MQ, Pub/Sub.

• Why Kafka instead of traditional ETL

Kafka enables continuous/event-driven processing. Kafka is primarily an event streaming and durable messaging platform. It can become part of a streaming ETL architecture, but transformation, validation, enrichment , deduplication, and loading are still performed by downstream processing systems such as Spark, Flink, Kafka Streams, or warehouse-native processing.

• Kafka Scaling

• Partitions A Topic isn't just one big file. To allow for parallel processing, Kafka splits a Topic into multiple Partitions. Think of a partition as a commit log where data is only appended at the end.This architecture enables multiple producers to write to different partitions simultaneously and multiple consumers to read from different partitions simultaneously.

• Consumer Groups To read partitions in parallel, consumers are organized into Consumer Groups. Kafka automatically ensures that each partition in a topic is assigned to exactly one consumer within that group. This provides massive scalability.

• Partitioning strategy

• The same key is normally mapped to the same partition as

• long as the partitioning configuration remains compatible.

• Ordering guarantees

• Kafka guarantees message ordering within a partition, not across an entire topic.

• Consumer groups

• A consumer group is a collection of Kafka consumers that work together to consume messages from a topic.

• Offset management

• An offset is the position of a message within a Kafka partition.
• Example:
  

• If a consumer has successfully processed:
  
  
  
  
• it can commit offset 3.
• The important concept is that the committed offset represents the next position to consume, depending on the consumer's offset semantics.

• Each partition is assigned to only one consumer within a consumer group at a time.

• This allows Kafka consumers to process data in parallel.

• Rebalancing
• A rebalance occurs when Kafka needs to redistribute partitions among consumers in a consumer group.

• During some rebalance scenarios, partition ownership changes and consumption can temporarily pause.

  Frequent rebalancing can therefore reduce throughput and increase latency.

  Common causes include:

  • Consumer crashes
  • Consumer restarts
  • New consumers joining
  • Consumers leaving
  • Changes in topic partition count
  • Slow processing causing consumer timeout/poll issues

  Practical example

  Suppose Spark is consuming Kafka data and one executor/consumer becomes unavailable.

  Kafka may rebalance the partitions among the remaining consumers.

  This is one reason consumer configuration and processing time are important in production systems.

• Replication factor

• Kafka provides fault tolerance through replication.

• Replication factor vs partitions

  These are different concepts.

  Partitions provide:

  • Parallelism
  • Scalability
  • Distribution

  Replication provides:

  • Fault tolerance
  • Availability
  • Data durability

• Producer acknowledgements

• Kafka producers use the acks configuration to control how strongly they want Kafka to acknowledge a write.

• Advantages:

  • Very low latency
  • High throughput

  Disadvantage:

  • Producer doesn't know whether Kafka successfully received the message.

  Potential message loss is therefore higher.

• Idempotent producer

• One problem in distributed systems is duplicate messages.
• Kafka receives the message successfully. But the acknowledgement gets lost:The producer doesn't know whether Kafka received it.

Now Kafka might receive the same logical message twice.


kafka provides an idempotent producer mechanism that allows retries without creating duplicate records in the partition log under the supported producer semantics.

• At-most-once vs at-least-once vs exactly-once

• This is one of the most important Kafka interview topics.

At-most-once

• The message is processed zero or one time.