Apache Kafka Architecture Overview

Kafka follows a publish-subscribe model where producers send messages to Kafka topics, and consumers read from these topics. Apache Kafka is a distributed event streaming platform widely used for building real-time data pipelines and event-driven architectures. It provides high-throughput, fault tolerance, and scalability, making it a go-to choice for microservices communication, logging, analytics, and more.

Producers: Services that create and send events.

Consumers: Services that read and process events.

Topics: A logical bucket or category for events.

Broker: The actual servers that store and manage the data. Unlike traditional message brokers, Kafka persists data to disk, allowing it to be re-read (which is not the case with Rabbit MQ, Active MQ, Pub/Sub.

• Why Kafka instead of traditional ETL

Kafka enables continuous/event-driven processing. Kafka is primarily an event streaming and durable messaging platform. It can become part of a streaming ETL architecture, but transformation, validation, enrichment , deduplication, and loading are still performed by downstream processing systems such as Spark, Flink, Kafka Streams, or warehouse-native processing.

• Kafka Scaling

• Partitions A Topic isn't just one big file. To allow for parallel processing, Kafka splits a Topic into multiple Partitions. Think of a partition as a commit log where data is only appended at the end.This architecture enables multiple producers to write to different partitions simultaneously and multiple consumers to read from different partitions simultaneously.

• Consumer Groups To read partitions in parallel, consumers are organized into Consumer Groups. Kafka automatically ensures that each partition in a topic is assigned to exactly one consumer within that group. This provides massive scalability.

• Partitioning strategy

• The same key is normally mapped to the same partition as

• long as the partitioning configuration remains compatible.

• Ordering guarantees

• Kafka guarantees message ordering within a partition, not across an entire topic.

• Consumer groups

• A consumer group is a collection of Kafka consumers that work together to consume messages from a topic.

• Offset management

• An offset is the position of a message within a Kafka partition.
• Example:
  

• If a consumer has successfully processed:
  
  
  
  
• it can commit offset 3.
• The important concept is that the committed offset represents the next position to consume, depending on the consumer's offset semantics.

• Each partition is assigned to only one consumer within a consumer group at a time.

• This allows Kafka consumers to process data in parallel.

• Rebalancing
• A rebalance occurs when Kafka needs to redistribute partitions among consumers in a consumer group.

• During some rebalance scenarios, partition ownership changes and consumption can temporarily pause.

  Frequent rebalancing can therefore reduce throughput and increase latency.

  Common causes include:

  • Consumer crashes
  • Consumer restarts
  • New consumers joining
  • Consumers leaving
  • Changes in topic partition count
  • Slow processing causing consumer timeout/poll issues

  Practical example

  Suppose Spark is consuming Kafka data and one executor/consumer becomes unavailable.

  Kafka may rebalance the partitions among the remaining consumers.

  This is one reason consumer configuration and processing time are important in production systems.

• Replication factor

• Kafka provides fault tolerance through replication.

• Replication factor vs partitions

  These are different concepts.

  Partitions provide:

  • Parallelism
  • Scalability
  • Distribution

  Replication provides:

  • Fault tolerance
  • Availability
  • Data durability

• Producer acknowledgements

• Kafka producers use the acks configuration to control how strongly they want Kafka to acknowledge a write.

• Advantages:

  • Very low latency
  • High throughput

  Disadvantage:

  • Producer doesn't know whether Kafka successfully received the message.

  Potential message loss is therefore higher.

• Idempotent producer

• One problem in distributed systems is duplicate messages.
• Kafka receives the message successfully. But the acknowledgement gets lost:The producer doesn't know whether Kafka received it.

Now Kafka might receive the same logical message twice.


kafka provides an idempotent producer mechanism that allows retries without creating duplicate records in the partition log under the supported producer semantics.

• At-most-once vs at-least-once vs exactly-once

• This is one of the most important Kafka interview topics.

At-most-once

• The message is processed zero or one time.

Big Data and Distributed Systems

Big Data refers to datasets that are too large, fast, or complex to be processed using traditional systems.

🔷 The 5 V’s of Big Data

Volume – How much data -> Refers to the huge amount of data generated and stored.

Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.

Variety – Different data types -> Refers to the different types and formats of data.

Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.

Value – Business usefulness -> Refers to the business insight or benefit extracted from data.

🔷Why RDBMS Fails

• Vertical scaling limit

• Expensive hardware

• Single point of failure

🔷How Big Data Systems Help

• Horizontal scaling

• Fault tolerance

• Distributed storage & processing

---

Distributed Infrastructure

A distributed system is a collection of independent computers that work together as a single system.

CAP Theorem

In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.

🔹 The 3 CAP Properties

Consistency (C)

All nodes see the same data at the same time.

📌 After a write, every read gets the latest value.

Availability (A)

Every request receives a response, even if it may not be the latest data.

📌 System is always responsive.

Network Partition Tolerance (P)

The system continues to operate despite network failures between nodes.

📌 Network partitions are unavoidable in distributed systems.

• Distributed Storage

• Communication

• Distributed Computation

Implementation:

        Remote Procedure Calls (RPC)

Threads
Concurrency Control

Performance:

Scalability (2 * computation power -> 2* throughput)

Computational Speed - reduced timing for programme execution 

Fault Tolerance:

Availability

Recoverability

Replication

Consistency:

Put Operation (Key, Value)

Get Operation (Key, value)

🔹 Rack

A rack is a physical group of nodes connected to the same network switch.

---

Hadoop

Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.

Components:

• HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.

• YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.

• MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.

• Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.

• Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
  “Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
  
  Example

• Replication factor = 3

• Each block is stored 3 times on 3 different DataNodes

So if you have a 128 MB block:

Block 1 (128 MB) → DN1 

Block 1 (128 MB) → DN2

Block 1 (128 MB) → DN3

• Why replication is needed

1️⃣ Fault tolerance - If one DataNode fails → data still available

2️⃣ High availability - Client can read from nearest replica

3️⃣ Better read performance - Parallel reads from different nodes

🧠 How replication works internally

• NameNode decides:

• Where replicas go

• Ensures replicas are on different nodes

• Uses rack awareness
  
  

If a DataNode fails:

• NameNode detects missing replica

• Creates new replica automatically
  
  

DataNodes send heartbeats to NameNode:

• Default: every 3 seconds

• If no heartbeat for 10 minutes → DataNode marked dead

HDFS must always maintain 3 healthy copies of every block (depends on replication factor)

• Why NameNode Creates a New Replica : 

• Replication factor is a minimum guarantee

• It is not “create once and forget”

• It must be maintained throughout cluster lifetime

🔹Self-Healing Mechanism (CORE HDFS FEATURE)

HDFS is designed to: Detect → Decide → Replicate → Recover

• Replica Creation Flow (IMPORTANT)

HDFS’s standard rack awareness policy dictates two main rules:

• No single rack should hold more than two replicas of the same block.

• Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.

• NameNode detects missing block replica

• Marks block as under-replicated

• Selects:

• Source DataNode (healthy replica)

• Target DataNode

• Must be:

• Alive

• Have enough disk space

• Follow rack-awareness rules

• Existing healthy DataNode is selected

• No new machine is created

• Orders DataNode to copy block

• Replication factor restored

• Happens automatically

• No user intervention

Steps:

1. NameNode sends metadata instruction (RPC) Remote Procedure Calls:
   DN1 replicate block B1 to DN4

2. DN1 opens a TCP connection to DN4

3. Data is transferred using DataTransferProtocol (using TCP)

4. Checksums are verified during transfer

5. DN4 confirms successful write

6. NameNode updates metadata

> Actual data never goes through the NameNode

• What is Rebalancer?

Rebalancer is a manual administrative tool used to:

• evenly distribute HDFS blocks across DataNodes.

🔸 When Is Rebalancer Used?

• New DataNodes added

• Some nodes are full

• Uneven disk utilization

🔸 Who Triggers It?

❌ Not automatic
✅ Admin manually runs it

🔸 Purpose

• Balance storage usage

• Improve performance

• Prevent hotspots

Question: What exactly happens when a SINGLE file is processed using Hadoop?

Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.

Answer: 

• File storage in HDFS (before processing) HDFS block-splitting
  Default HDFS block size = 128 MB (sometimes 64 MB)
  A 120 GB file ≈ 120 × 1024 = 122,880 MB
  
  

Number of blocks:

122,880 MB / 128 MB ≈ 960 blocks

The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies

• Job submission (MapReduce)
  Client submit jobs to YARN ResourceManager.
  ResourceManager asks NameNode: Where are the blocks stored?
  Jobs are divided into tasks.
  
  

• Map phase (parallel processing)
  One Mapper per block (usually)

Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks

{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,

Containers define how many can run at once. }

Question: What happens if one DataNode goes down?
Answer: 

Scenario

• You have 5 DataNodes

• Replication factor = 3

• One DataNode crashes during job execution

NameNode detects failure

• DataNodes send heartbeat every ~3 seconds

• If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD

HDFS handles data automatically

• Blocks on the dead DataNode are now under-replicated

• NameNode schedules re-replication:

• Copies missing blocks from healthy nodes

• Restores replication factor

👉 No data loss

Running Map tasks on that node

• Any mapper running on the failed node fails

• YARN:

• Reschedules the same map task

• Runs it on another node that has a replica of the block

👉 Only failed tasks rerun, not the whole job

Reduce tasks

• Reduce tasks are not tied to data locality

• They continue normally unless the node running the reducer fails

Question: How to calculate optimal HDFS block size?

Answer:

Default block size

• 128 MB (modern clusters)

• 64 MB (older)

Balance between:

• Parallelism

• Task overhead

• Disk & network efficiency

Use large blocks when:

• Large files (GBs/TBs)

• Sequential reads

• Batch analytics

Use small blocks when:

• Many small files

• Low latency needed

Practical formula (industry-used)

• Aim for 1–2 minutes per mapper

For your 120 GB file

Block size	No of Blocks	Mapper waves (5 nodes)
128 MB	~960	Too many
256 MB	~480	Better
512 MB	~240	Best
1 GB	~120	Good if memory allows

We can provide custom value in $HADOOP_HOME/hadoop*.xml file:

<property>

  <name>dfs.blocksize</name>

  <value>268435456</value> <!-- 256 MB -->

</property>

---

🐘 Hadoop Architecture (Big Picture)

• Hadoop has 3 core layers:

1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.

Architecture Components

• NameNode (Master)

What it does

• Stores metadata only (in memory):

• File names

• Block IDs

• Block locations

• Permissions

• Decides:

• Where blocks go

• From where to read data

What it does NOT do

                ❌ Store actual data

                ❌ Handle client reads/writes directly

• DataNode (Worker)

What it does

• Stores actual data blocks

• Sends heartbeats to NameNode

• Sends block reports

On failure - NameNode re-replicates blocks

• Secondary NameNode (Checkpoint Node) - NOT a backup NameNode

• Purpose Periodically:

• Merges fsimage + edits

• Creates checkpoint

 Reduces NameNode recovery time.  Cannot take over automatically.

• FSImage in HDFS:

                        1️⃣ What is FSImage?

             FSImage is a persistent snapshot of the HDFS namespace metadata stored on                         disk.

It contains:

• Directory structure

• File names

• Permissions

• Block mapping (file → block IDs)

It does NOT contain actual data blocks.

2️⃣Why FSImage Is Needed

NameNode metadata is stored in memory for fast access.
FSImage ensures:

• Metadata survives NameNode restart

• Faster recovery after failure

                3️⃣ What FSImage Contains

 ✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files

 ❌ Block locations (DataNode info)
❌ Actual data blocks

 📌 Block locations are provided by DataNodes via heartbeats.

• Standby NameNode (HA)

Used in High Availability setup

✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination

• What is ZooKeeper?

Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:

• Configuration

• Naming

• Synchronization

• Leader election

• Failure detection

📌 It does NOT store application data
📌 It stores small metadata only

• MapReduce (Built in 2004) - Indexing the webpages is equivalent to sorting of webpages.
  
  

MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.

MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.

Why MapReduce:

Huge amounts of data were stored in single servers prior to 2004.

The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.

With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.

MapReduce Word Count Problem:

• Input

• Splitting

• Mapping

• Shuffling

• Reducing

Map Execution Phase

• MapPhase

Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.

• Partition Phase

Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.

• Shuffle Phase

During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.

• Sort Phase

The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.

• Reduce Phase

The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.

Execution of MapReduce:

• GFS (Google File System) / HDFS (Hadoop Distributed File System):

These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS). 

By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.

Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.

For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB. 

Total blocks = 100 GB / 1 GB = 100 blocks

• HDFS does not assign blocks per node upfront.

• Blocks are placed individually, not node-wise.

Some nodes might have 8 blocks, some 12, some 15, etc.

For replication factor = 5:

• 1 replica on local rack

• 2 replicas on a different rack

• Remaining replicas distributed to balance load

If a node fails:

• Tasks running on that node are rescheduled

• Data is read from replica blocks on other nodes

• No block is “lost” logically

• NameNode detects under-replicated blocks and re-replicates them.

Question: What Shuffle Actually Does?

Answer: Shuffle is responsible for:

1. Partitioning – deciding which reducer gets which keys

2. Sorting – sorting intermediate keys

3. Grouping – grouping all values for the same key

Mapper Output:

(word, 1)

(word, 1)

(hadoop, 1)

(word, 1)

After Shuffle (Reducer Input):

(word, [1, 1, 1])

(hadoop, [1])

👉 All values of the same key are sent to the same reducer.

Question : What is a Reducer in Hadoop MapReduce?

Answer: A reducer takes a key and all its associated values and combines them to produce final output records.

A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.

YARN:Yet Another Resource Negotiator

1️⃣ What is YARN?

YARN is Hadoop’s cluster resource management and job scheduling layer.

 📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez

2️⃣ Why YARN Was Introduced

Problem before YARN (Hadoop 1.x)

• MapReduce JobTracker:

• Resource management

• Job scheduling

• Job monitoring

• Single point of failure

• Scalability limits

Solution -  YARN separates resource management from processing

3️⃣ Core Components of YARN

🔹 1. ResourceManager (RM)

• Global master

• Allocates cluster resources

• Does NOT execute tasks

Sub-components:

• Scheduler (FIFO / Capacity / Fair) - 3 types of schedulers - Use for allocating the resources
  FIFO and Fair are Spark job scheduling modes.
  Capacity Scheduler is a YARN cluster scheduler, not a Spark scheduler.

Feature	FIFO (Spark)	Fair (Spark)	Capacity (YARN)
Scope	Single Spark application	Single Spark application	Entire cluster
Scheduling	First submitted job first	Resources shared fairly	Queue-based resource allocation
Multi-user support	Poor	Good	Excellent
Starvation	Possible	Much less likely	Prevented through queue guarantees
Best for	Development, single workload	Shared Spark environments	Enterprise clusters

✅ Scheduler

• Decides how many containers

• Decides which NodeManager

• Decides resource size (CPU, memory)

• Scheduler only allocates containers logically

• It does NOT create containers physically
  
  

• ApplicationsManager
  

🔹 2. NodeManager (NM)

• Runs on every node

• Manages containers

• Monitors resource usage

• Reports to ResourceManager

✅ NodeManager

• Receives allocation info from RM

• Creates the container

• Launches the process

• Monitors resource usage

• Kills container if limits exceeded

• Sends heartbeat to RM

• Cleans up after completion

📌 Container = Linux process started by NodeManager

🔹 3. ApplicationMaster (AM)

• One per application

• Negotiates resources with RM

• Manages application lifecycle

📌 Mandatory for every YARN app

🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.

• Resource unit

• Includes:

• Memory

• CPU

• Disk

• Network

📌 Tasks run inside containers

🔄 End-to-End Flow (Very Important)

1. ApplicationMaster requests containers

2. ResourceManager Scheduler allocates containers

3. NodeManager creates the container

4. NodeManager runs & monitors the container

5. NodeManager releases resources after completion

4️⃣ YARN Execution Flow (Step-by-Step)

Example: MapReduce Job

1. Client submits job to RM

2. RM launches ApplicationMaster

3. AM registers with RM

4. AM requests containers

5. RM allocates containers

6. NMs launch containers

7. Tasks execute

8. AM reports status

9. Job completes, resources released

5️⃣ YARN Scheduling Policies

Big Data 

Big Data refers to datasets that are too large, fast, or complex to be processed using traditional systems.

🔷 The 5 V’s of Big Data

Volume – How much data -> Refers to the huge amount of data generated and stored.

Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.

Variety – Different data types -> Refers to the different types and formats of data.

Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.

Value – Business usefulness -> Refers to the business insight or benefit extracted from data.

🔷Why RDBMS Fails

• Vertical scaling limit

• Expensive hardware

• Single point of failure

🔷How Big Data Systems Help

• Horizontal scaling

• Fault tolerance

• Distributed storage & processing

---

Distributed Infrastructure

A distributed system is a collection of independent computers that work together as a single system.

CAP Theorem

In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.

🔹 The 3 CAP Properties

Consistency (C)

All nodes see the same data at the same time.

📌 After a write, every read gets the latest value.

Availability (A)

Every request receives a response, even if it may not be the latest data.

📌 System is always responsive.

Network Partition Tolerance (P)

The system continues to operate despite network failures between nodes.

📌 Network partitions are unavoidable in distributed systems.

• Distributed Storage

• Communication

• Distributed Computation

Implementation:

Remote Procedure Calls (RPC)

Threads
Concurrency Control

Performance:

Scalability (2 * computation power -> 2* throughput)

Computational Speed - reduced timing for programme execution 

Fault Tolerance:

Availability

Recoverability

Replication

Consistency:

Put Operation (Key, Value)

Get Operation (Key, value)

🔹 Rack

A rack is a physical group of nodes connected to the same network switch.

---

Hadoop

Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.

Components:

• HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.

• YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.

• MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.

• Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.

• Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
  “Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
  
  Example

• Replication factor = 3

• Each block is stored 3 times on 3 different DataNodes

So if you have a 128 MB block:

Block 1 (128 MB) → DN1 

Block 1 (128 MB) → DN2

Block 1 (128 MB) → DN3

• Why replication is needed

1️⃣ Fault tolerance - If one DataNode fails → data still available

2️⃣ High availability - Client can read from nearest replica

3️⃣ Better read performance - Parallel reads from different nodes

🧠 How replication works internally

• NameNode decides:

• Where replicas go

• Ensures replicas are on different nodes

• Uses rack awareness
  
  

If a DataNode fails:

• NameNode detects missing replica

• Creates new replica automatically
  
  

DataNodes send heartbeats to NameNode:

• Default: every 3 seconds

• If no heartbeat for 10 minutes → DataNode marked dead

HDFS must always maintain 3 healthy copies of every block (depends on replication factor)

• Why NameNode Creates a New Replica : 

• Replication factor is a minimum guarantee

• It is not “create once and forget”

• It must be maintained throughout cluster lifetime

🔹Self-Healing Mechanism (CORE HDFS FEATURE)

HDFS is designed to: Detect → Decide → Replicate → Recover

• Replica Creation Flow (IMPORTANT)

HDFS’s standard rack awareness policy dictates two main rules:

• No single rack should hold more than two replicas of the same block.

• Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.

• NameNode detects missing block replica

• Marks block as under-replicated

• Selects:

• Source DataNode (healthy replica)

• Target DataNode

• Must be:

• Alive

• Have enough disk space

• Follow rack-awareness rules

• Existing healthy DataNode is selected

• No new machine is created

• Orders DataNode to copy block

• Replication factor restored

• Happens automatically

• No user intervention

Steps:

1. NameNode sends metadata instruction (RPC) Remote Procedure Calls:
   DN1 replicate block B1 to DN4

2. DN1 opens a TCP connection to DN4

3. Data is transferred using DataTransferProtocol (using TCP)

4. Checksums are verified during transfer

5. DN4 confirms successful write

6. NameNode updates metadata

> Actual data never goes through the NameNode

• What is Rebalancer?

Rebalancer is a manual administrative tool used to:

• evenly distribute HDFS blocks across DataNodes.

🔸 When Is Rebalancer Used?

• New DataNodes added

• Some nodes are full

• Uneven disk utilization

🔸 Who Triggers It?

❌ Not automatic
✅ Admin manually runs it

🔸 Purpose

• Balance storage usage

• Improve performance

• Prevent hotspots

Question: What exactly happens when a SINGLE file is processed using Hadoop?

Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.

Answer: 

• File storage in HDFS (before processing) HDFS block-splitting
  Default HDFS block size = 128 MB (sometimes 64 MB)
  A 120 GB file ≈ 120 × 1024 = 122,880 MB
  
  

Number of blocks:

122,880 MB / 128 MB ≈ 960 blocks

The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies

• Job submission (MapReduce)
  Client submit jobs to YARN ResourceManager.
  ResourceManager asks NameNode: Where are the blocks stored?
  Jobs are divided into tasks.
  
  

• Map phase (parallel processing)
  One Mapper per block (usually)

Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks

{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,

Containers define how many can run at once. }

Question: What happens if one DataNode goes down?
Answer: 

Scenario

• You have 5 DataNodes

• Replication factor = 3

• One DataNode crashes during job execution

NameNode detects failure

• DataNodes send heartbeat every ~3 seconds

• If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD

HDFS handles data automatically

• Blocks on the dead DataNode are now under-replicated

• NameNode schedules re-replication:

• Copies missing blocks from healthy nodes

• Restores replication factor

👉 No data loss

Running Map tasks on that node

• Any mapper running on the failed node fails

• YARN:

• Reschedules the same map task

• Runs it on another node that has a replica of the block

👉 Only failed tasks rerun, not the whole job

Reduce tasks

• Reduce tasks are not tied to data locality

• They continue normally unless the node running the reducer fails

Question: How to calculate optimal HDFS block size?

Answer:

Default block size

• 128 MB (modern clusters)

• 64 MB (older)

Balance between:

• Parallelism

• Task overhead

• Disk & network efficiency

Use large blocks when:

• Large files (GBs/TBs)

• Sequential reads

• Batch analytics

Use small blocks when:

• Many small files

• Low latency needed

Practical formula (industry-used)

• Aim for 1–2 minutes per mapper

For your 120 GB file

Block size	No of Blocks	Mapper waves (5 nodes)
128 MB	~960	Too many
256 MB	~480	Better
512 MB	~240	Best
1 GB	~120	Good if memory allows

We can provide custom value in $HADOOP_HOME/hadoop*.xml file:

<property>

  <name>dfs.blocksize</name>

  <value>268435456</value> <!-- 256 MB -->

</property>

---

🐘 Hadoop Architecture (Big Picture)

• Hadoop has 3 core layers:

1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.

Architecture Components

• NameNode (Master)

What it does

• Stores metadata only (in memory):

• File names

• Block IDs

• Block locations

• Permissions

• Decides:

• Where blocks go

• From where to read data

What it does NOT do

❌ Store actual data
❌ Handle client reads/writes directly

• DataNode (Worker)

What it does

• Stores actual data blocks

• Sends heartbeats to NameNode

• Sends block reports

On failure - NameNode re-replicates blocks

• Secondary NameNode (Checkpoint Node) - NOT a backup NameNode

• Purpose Periodically:

• Merges fsimage + edits

• Creates checkpoint

 Reduces NameNode recovery time.  Cannot take over automatically.

• FSImage in HDFS:

1️⃣ What is FSImage?

FSImage is a persistent snapshot of the HDFS namespace metadata stored on disk.

It contains:

• Directory structure

• File names

• Permissions

• Block mapping (file → block IDs)

It does NOT contain actual data blocks.

2️⃣ Why FSImage Is Needed

NameNode metadata is stored in memory for fast access.
FSImage ensures:

• Metadata survives NameNode restart

• Faster recovery after failure

3️⃣ What FSImage Contains

 ✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files

 ❌ Block locations (DataNode info)
❌ Actual data blocks

 📌 Block locations are provided by DataNodes via heartbeats.

• Standby NameNode (HA)

Used in High Availability setup

✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination

• What is ZooKeeper?

Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:

• Configuration

• Naming

• Synchronization

• Leader election

• Failure detection

📌 It does NOT store application data
📌 It stores small metadata only

• MapReduce (Built in 2004) - Indexing the webpages is equivalent to sorting of webpages.
  
  

MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.

MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.

Why MapReduce:

Huge amounts of data were stored in single servers prior to 2004.

The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.

With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.

MapReduce Word Count Problem:

• Input

• Splitting

• Mapping

• Shuffling

• Reducing

Map Execution Phase

• MapPhase

Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.

• Partition Phase

Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.

• Shuffle Phase

During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.

• Sort Phase

The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.

• Reduce Phase

The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.

Execution of MapReduce:

• GFS (Google File System) / HDFS (Hadoop Distributed File System):

These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS). 

By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.

Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.

For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB. 

Total blocks = 100 GB / 1 GB = 100 blocks

• HDFS does not assign blocks per node upfront.

• Blocks are placed individually, not node-wise.

Some nodes might have 8 blocks, some 12, some 15, etc.

For replication factor = 5:

• 1 replica on local rack

• 2 replicas on a different rack

• Remaining replicas distributed to balance load

If a node fails:

• Tasks running on that node are rescheduled

• Data is read from replica blocks on other nodes

• No block is “lost” logically

• NameNode detects under-replicated blocks and re-replicates them.

Question: What Shuffle Actually Does?

Answer: Shuffle is responsible for:

1. Partitioning – deciding which reducer gets which keys

2. Sorting – sorting intermediate keys

3. Grouping – grouping all values for the same key

Mapper Output:

(word, 1)

(word, 1)

(hadoop, 1)

(word, 1)

After Shuffle (Reducer Input):

(word, [1, 1, 1])

(hadoop, [1])

👉 All values of the same key are sent to the same reducer.

Question : What is a Reducer in Hadoop MapReduce?

Answer: 

A reducer takes a key and all its associated values and combines them to produce final output records.

A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.

YARN:Yet Another Resource Negotiator

1️⃣ What is YARN?

YARN is Hadoop’s cluster resource management and job scheduling layer.

 📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez

2️⃣ Why YARN Was Introduced

Problem before YARN (Hadoop 1.x)

• MapReduce JobTracker:

• Resource management

• Job scheduling

• Job monitoring

• Single point of failure

• Scalability limits

Solution -  YARN separates resource management from processing

3️⃣ Core Components of YARN

🔹 1. ResourceManager (RM)

• Global master

• Allocates cluster resources

• Does NOT execute tasks

Sub-components:

• Scheduler (FIFO / Capacity / Fair) - 3 types of schedulers - Use for allocating the resources
  FIFO and Fair are Spark job scheduling modes.
  Capacity Scheduler is a YARN cluster scheduler, not a Spark scheduler.

Feature	FIFO (Spark)	Fair (Spark)	Capacity (YARN)
Scope	Single Spark application	Single Spark application	Entire cluster
Scheduling	First submitted job first	Resources shared fairly	Queue-based resource allocation
Multi-user support	Poor	Good	Excellent
Starvation	Possible	Much less likely	Prevented through queue guarantees
Best for	Development, single workload	Shared Spark environments	Enterprise clusters

✅ Scheduler

• Decides how many containers

• Decides which NodeManager

• Decides resource size (CPU, memory)

• Scheduler only allocates containers logically

• It does NOT create containers physically
  
  

• ApplicationsManager
  

🔹 2. NodeManager (NM)

• Runs on every node

• Manages containers

• Monitors resource usage

• Reports to ResourceManager

✅ NodeManager

• Receives allocation info from RM

• Creates the container

• Launches the process

• Monitors resource usage

• Kills container if limits exceeded

• Sends heartbeat to RM

• Cleans up after completion

📌 Container = Linux process started by NodeManager

🔹 3. ApplicationMaster (AM)

• One per application

• Negotiates resources with RM

• Manages application lifecycle

📌 Mandatory for every YARN app

🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.

• Resource unit

• Includes:

• Memory

• CPU

• Disk

• Network

📌 Tasks run inside containers

🔄 End-to-End Flow (Very Important)

1. ApplicationMaster requests containers

2. ResourceManager Scheduler allocates containers

3. NodeManager creates the container

4. NodeManager runs & monitors the container

5. NodeManager releases resources after completion

4️⃣ YARN Execution Flow (Step-by-Step)

Example: MapReduce Job

1. Client submits job to RM

2. RM launches ApplicationMaster

3. AM registers with RM

4. AM requests containers

5. RM allocates containers

6. NMs launch containers

7. Tasks execute

8. AM reports status

9. Job completes, resources released

5️⃣ YARN Scheduling Policies

6️⃣ YARN High Availability

• Multiple ResourceManagers

• ZooKeeper for:

• Leader election

• Failover

7️⃣ What YARN Does NOT Do ❌

 ❌ Store data
❌ Execute tasks itself
❌ Replace HDFS

8️⃣ YARN vs MapReduce

🧾 Summary

• YARN separates resource management from data processing.

• ResourceManager allocates resources but does not execute tasks.

• Every YARN application has one ApplicationMaster.

• Containers are the execution units in YARN.

• YARN = cluster resource manager

• Enables multi-engine Hadoop

• Improves scalability & reliability

• Backbone of modern Hadoop

6️⃣ YARN High Availability

• Multiple ResourceManagers

• ZooKeeper for:

• Leader election

• Failover

7️⃣ What YARN Does NOT Do ❌

 ❌ Store data
❌ Execute tasks itself
❌ Replace HDFS

8️⃣ YARN vs MapReduce

🧾 Summary

• YARN separates resource management from data processing.

• ResourceManager allocates resources but does not execute tasks.

• Every YARN application has one ApplicationMaster.

• Containers are the execution units in YARN.

• YARN = cluster resource manager

• Enables multi-engine Hadoop

• Improves scalability & reliability

• Backbone of modern Hadoop