Big Data refers to datasets that are too large, fast, or complex to be processed using traditional systems.
🔷 The 5 V’s of Big Data
Volume – How much data -> Refers to the huge amount of data generated and stored.
Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.
Variety – Different data types -> Refers to the different types and formats of data.
Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.
Value – Business usefulness -> Refers to the business insight or benefit extracted from data.
🔷Why RDBMS Fails
Vertical scaling limit
Expensive hardware
Single point of failure
🔷How Big Data Systems Help
Horizontal scaling
Fault tolerance
Distributed storage & processing
Distributed Infrastructure
A distributed system is a collection of independent computers that work together as a single system.
CAP Theorem
In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.
🔹 The 3 CAP Properties
Consistency (C)
All nodes see the same data at the same time.
📌 After a write, every read gets the latest value.
Availability (A)
Every request receives a response, even if it may not be the latest data.
📌 System is always responsive.
Network Partition Tolerance (P)
The system continues to operate despite network failures between nodes.
📌 Network partitions are unavoidable in distributed systems.
Distributed Storage
Communication
Distributed Computation
Implementation:
Remote Procedure Calls (RPC)
Threads
Concurrency Control
Performance:
Scalability (2 * computation power -> 2* throughput)
Computational Speed - reduced timing for programme execution
Fault Tolerance:
Availability
Recoverability
Replication
Consistency:
Put Operation (Key, Value)
Get Operation (Key, value)
🔹 Rack
A rack is a physical group of nodes connected to the same network switch.
Hadoop
Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.
Components:
HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.
YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.
MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.
Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.
Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
“Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
ExampleReplication factor = 3
Each block is stored 3 times on 3 different DataNodes
So if you have a 128 MB block:
Block 1 (128 MB) → DN1
Block 1 (128 MB) → DN2
Block 1 (128 MB) → DN3
Why replication is needed
1️⃣ Fault tolerance - If one DataNode fails → data still available
2️⃣ High availability - Client can read from nearest replica
3️⃣ Better read performance - Parallel reads from different nodes
🧠 How replication works internally
NameNode decides:
Where replicas go
Ensures replicas are on different nodes
Uses rack awareness
If a DataNode fails:
NameNode detects missing replica
Creates new replica automatically
DataNodes send heartbeats to NameNode:
Default: every 3 seconds
If no heartbeat for 10 minutes → DataNode marked dead
HDFS must always maintain 3 healthy copies of every block (depends on replication factor)
Why NameNode Creates a New Replica :
Replication factor is a minimum guarantee
It is not “create once and forget”
It must be maintained throughout cluster lifetime
🔹Self-Healing Mechanism (CORE HDFS FEATURE)
HDFS is designed to: Detect → Decide → Replicate → Recover
Replica Creation Flow (IMPORTANT)
HDFS’s standard rack awareness policy dictates two main rules:
No single rack should hold more than two replicas of the same block.
Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.
NameNode detects missing block replica
Marks block as under-replicated
Selects:
Source DataNode (healthy replica)
Target DataNode
Must be:
Alive
Have enough disk space
Follow rack-awareness rules
Existing healthy DataNode is selected
No new machine is created
Orders DataNode to copy block
Replication factor restored
Happens automatically
No user intervention
Steps:
NameNode sends metadata instruction (RPC) Remote Procedure Calls:
DN1 replicate block B1 to DN4DN1 opens a TCP connection to DN4
Data is transferred using DataTransferProtocol (using TCP)
Checksums are verified during transfer
DN4 confirms successful write
NameNode updates metadata
> Actual data never goes through the NameNode
What is Rebalancer?
Rebalancer is a manual administrative tool used to:
evenly distribute HDFS blocks across DataNodes.
🔸 When Is Rebalancer Used?
New DataNodes added
Some nodes are full
Uneven disk utilization
🔸 Who Triggers It?
❌ Not automatic
✅ Admin manually runs it
🔸 Purpose
Balance storage usage
Improve performance
Prevent hotspots
Question: What exactly happens when a SINGLE file is processed using Hadoop?
Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.
Answer:
File storage in HDFS (before processing) HDFS block-splitting
Default HDFS block size = 128 MB (sometimes 64 MB)
A 120 GB file ≈ 120 × 1024 = 122,880 MB
Number of blocks:
122,880 MB / 128 MB ≈ 960 blocks
The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies
Job submission (MapReduce)
Client submit jobs to YARN ResourceManager.
ResourceManager asks NameNode: Where are the blocks stored?
Jobs are divided into tasks.Map phase (parallel processing)
One Mapper per block (usually)
Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks
{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,
Containers define how many can run at once. }
Question: What happens if one DataNode goes down?
Answer:
Scenario
You have 5 DataNodes
Replication factor = 3
One DataNode crashes during job execution
NameNode detects failure
DataNodes send heartbeat every ~3 seconds
If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD
HDFS handles data automatically
Blocks on the dead DataNode are now under-replicated
NameNode schedules re-replication:
Copies missing blocks from healthy nodes
Restores replication factor
👉 No data loss
Running Map tasks on that node
Any mapper running on the failed node fails
YARN:
Reschedules the same map task
Runs it on another node that has a replica of the block
👉 Only failed tasks rerun, not the whole job
Reduce tasks
Reduce tasks are not tied to data locality
They continue normally unless the node running the reducer fails
Question: How to calculate optimal HDFS block size?
Answer:
Default block size
128 MB (modern clusters)
64 MB (older)
Balance between:
Parallelism
Task overhead
Disk & network efficiency
Use large blocks when:
Large files (GBs/TBs)
Sequential reads
Batch analytics
Use small blocks when:
Many small files
Low latency needed
Practical formula (industry-used)
Aim for 1–2 minutes per mapper
For your 120 GB file
We can provide custom value in $HADOOP_HOME/hadoop*.xml file:
<property>
<name>dfs.blocksize</name>
<value>268435456</value> <!-- 256 MB -->
</property>
🐘 Hadoop Architecture (Big Picture)
Hadoop has 3 core layers:
1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.
Architecture Components
NameNode (Master)
What it does
Stores metadata only (in memory):
File names
Block IDs
Block locations
Permissions
Decides:
Where blocks go
From where to read data
What it does NOT do
❌ Store actual data
❌ Handle client reads/writes directly
DataNode (Worker)
What it does
Stores actual data blocks
Sends heartbeats to NameNode
Sends block reports
On failure - NameNode re-replicates blocks
Secondary NameNode (Checkpoint Node) - NOT a backup NameNode
Purpose Periodically:
Merges fsimage + edits
Creates checkpoint
Reduces NameNode recovery time. Cannot take over automatically.
FSImage in HDFS:
1️⃣ What is FSImage?
FSImage is a persistent snapshot of the HDFS namespace metadata stored on disk.
It contains:
Directory structure
File names
Permissions
Block mapping (file → block IDs)
It does NOT contain actual data blocks.
2️⃣Why FSImage Is Needed
NameNode metadata is stored in memory for fast access.
FSImage ensures:
Metadata survives NameNode restart
Faster recovery after failure
3️⃣ What FSImage Contains
✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files
❌ Block locations (DataNode info)
❌ Actual data blocks
📌 Block locations are provided by DataNodes via heartbeats.
Standby NameNode (HA)
Used in High Availability setup
✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination
What is ZooKeeper?
Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:
Configuration
Naming
Synchronization
Leader election
Failure detection
📌 It does NOT store application data
📌 It stores small metadata only
MapReduce (Built in 2004) - Indexing the webpages is equivalent to sorting of webpages.
MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.
MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.
Why MapReduce:
Huge amounts of data were stored in single servers prior to 2004.
The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.
With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.
MapReduce Word Count Problem:
Input
Splitting
Mapping
Shuffling
Reducing
Map Execution Phase
MapPhase
Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.
Partition Phase
Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.
Shuffle Phase
During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.
Sort Phase
The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.
Reduce Phase
The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.
Execution of MapReduce:
GFS (Google File System) / HDFS (Hadoop Distributed File System):
These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS).
By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.
Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.
For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB.
Total blocks = 100 GB / 1 GB = 100 blocks
HDFS does not assign blocks per node upfront.
Blocks are placed individually, not node-wise.
Some nodes might have 8 blocks, some 12, some 15, etc.
For replication factor = 5:
1 replica on local rack
2 replicas on a different rack
Remaining replicas distributed to balance load
If a node fails:
Tasks running on that node are rescheduled
Data is read from replica blocks on other nodes
No block is “lost” logically
NameNode detects under-replicated blocks and re-replicates them.
Question: What Shuffle Actually Does?
Answer: Shuffle is responsible for:
Partitioning – deciding which reducer gets which keys
Sorting – sorting intermediate keys
Grouping – grouping all values for the same key
Mapper Output:
(word, 1)
(word, 1)
(hadoop, 1)
(word, 1)
After Shuffle (Reducer Input):
(word, [1, 1, 1])
(hadoop, [1])
👉 All values of the same key are sent to the same reducer.
Question : What is a Reducer in Hadoop MapReduce?
Answer: A reducer takes a key and all its associated values and combines them to produce final output records.
A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.
YARN:Yet Another Resource Negotiator
1️⃣ What is YARN?
YARN is Hadoop’s cluster resource management and job scheduling layer.
📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez
2️⃣ Why YARN Was Introduced
Problem before YARN (Hadoop 1.x)
MapReduce JobTracker:
Resource management
Job scheduling
Job monitoring
Single point of failure
Scalability limits
Solution - YARN separates resource management from processing
3️⃣ Core Components of YARN
🔹 1. ResourceManager (RM)
Global master
Allocates cluster resources
Does NOT execute tasks
Sub-components:
Scheduler (FIFO / Capacity / Fair) - 3 types of schedulers - Use for allocating the resources
FIFO and Fair are Spark job scheduling modes.
Capacity Scheduler is a YARN cluster scheduler, not a Spark scheduler.
✅ Scheduler
Decides how many containers
Decides which NodeManager
Decides resource size (CPU, memory)
Scheduler only allocates containers logically
It does NOT create containers physically
ApplicationsManager
🔹 2. NodeManager (NM)
Runs on every node
Manages containers
Monitors resource usage
Reports to ResourceManager
✅ NodeManager
Receives allocation info from RM
Creates the container
Launches the process
Monitors resource usage
Kills container if limits exceeded
Sends heartbeat to RM
Cleans up after completion
📌 Container = Linux process started by NodeManager
🔹 3. ApplicationMaster (AM)
One per application
Negotiates resources with RM
Manages application lifecycle
📌 Mandatory for every YARN app
🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.
Resource unit
Includes:
Memory
CPU
Disk
Network
📌 Tasks run inside containers
🔄 End-to-End Flow (Very Important)
ApplicationMaster requests containers
ResourceManager Scheduler allocates containers
NodeManager creates the container
NodeManager runs & monitors the container
NodeManager releases resources after completion
4️⃣ YARN Execution Flow (Step-by-Step)
Example: MapReduce Job
Client submits job to RM
RM launches ApplicationMaster
AM registers with RM
AM requests containers
RM allocates containers
NMs launch containers
Tasks execute
AM reports status
Job completes, resources released
5️⃣ YARN Scheduling Policies
Big Data
Big Data refers to datasets that are too large, fast, or complex to be processed using traditional systems.
🔷 The 5 V’s of Big Data
Volume – How much data -> Refers to the huge amount of data generated and stored.
Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.
Variety – Different data types -> Refers to the different types and formats of data.
Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.
Value – Business usefulness -> Refers to the business insight or benefit extracted from data.
🔷Why RDBMS Fails
Vertical scaling limit
Expensive hardware
Single point of failure
🔷How Big Data Systems Help
Horizontal scaling
Fault tolerance
Distributed storage & processing
Distributed Infrastructure
A distributed system is a collection of independent computers that work together as a single system.
CAP Theorem
In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.
🔹 The 3 CAP Properties
Consistency (C)
All nodes see the same data at the same time.
📌 After a write, every read gets the latest value.
Availability (A)
Every request receives a response, even if it may not be the latest data.
📌 System is always responsive.
Network Partition Tolerance (P)
The system continues to operate despite network failures between nodes.
📌 Network partitions are unavoidable in distributed systems.
Distributed Storage
Communication
Distributed Computation
Implementation:
Remote Procedure Calls (RPC)
Threads
Concurrency Control
Performance:
Scalability (2 * computation power -> 2* throughput)
Computational Speed - reduced timing for programme execution
Fault Tolerance:
Availability
Recoverability
Replication
Consistency:
Put Operation (Key, Value)
Get Operation (Key, value)
🔹 Rack
A rack is a physical group of nodes connected to the same network switch.
Hadoop
Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.
Components:
HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.
YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.
MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.
Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.
Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
“Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
ExampleReplication factor = 3
Each block is stored 3 times on 3 different DataNodes
So if you have a 128 MB block:
Block 1 (128 MB) → DN1
Block 1 (128 MB) → DN2
Block 1 (128 MB) → DN3
Why replication is needed
1️⃣ Fault tolerance - If one DataNode fails → data still available
2️⃣ High availability - Client can read from nearest replica
3️⃣ Better read performance - Parallel reads from different nodes
🧠 How replication works internally
NameNode decides:
Where replicas go
Ensures replicas are on different nodes
Uses rack awareness
If a DataNode fails:
NameNode detects missing replica
Creates new replica automatically
DataNodes send heartbeats to NameNode:
Default: every 3 seconds
If no heartbeat for 10 minutes → DataNode marked dead
HDFS must always maintain 3 healthy copies of every block (depends on replication factor)
Why NameNode Creates a New Replica :
Replication factor is a minimum guarantee
It is not “create once and forget”
It must be maintained throughout cluster lifetime
🔹Self-Healing Mechanism (CORE HDFS FEATURE)
HDFS is designed to: Detect → Decide → Replicate → Recover
Replica Creation Flow (IMPORTANT)
HDFS’s standard rack awareness policy dictates two main rules:
No single rack should hold more than two replicas of the same block.
Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.
NameNode detects missing block replica
Marks block as under-replicated
Selects:
Source DataNode (healthy replica)
Target DataNode
Must be:
Alive
Have enough disk space
Follow rack-awareness rules
Existing healthy DataNode is selected
No new machine is created
Orders DataNode to copy block
Replication factor restored
Happens automatically
No user intervention
Steps:
NameNode sends metadata instruction (RPC) Remote Procedure Calls:
DN1 replicate block B1 to DN4DN1 opens a TCP connection to DN4
Data is transferred using DataTransferProtocol (using TCP)
Checksums are verified during transfer
DN4 confirms successful write
NameNode updates metadata
> Actual data never goes through the NameNode
What is Rebalancer?
Rebalancer is a manual administrative tool used to:
evenly distribute HDFS blocks across DataNodes.
🔸 When Is Rebalancer Used?
New DataNodes added
Some nodes are full
Uneven disk utilization
🔸 Who Triggers It?
❌ Not automatic
✅ Admin manually runs it
🔸 Purpose
Balance storage usage
Improve performance
Prevent hotspots
Question: What exactly happens when a SINGLE file is processed using Hadoop?
Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.
Answer:
File storage in HDFS (before processing) HDFS block-splitting
Default HDFS block size = 128 MB (sometimes 64 MB)
A 120 GB file ≈ 120 × 1024 = 122,880 MB
Number of blocks:
122,880 MB / 128 MB ≈ 960 blocks
The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies
Job submission (MapReduce)
Client submit jobs to YARN ResourceManager.
ResourceManager asks NameNode: Where are the blocks stored?
Jobs are divided into tasks.Map phase (parallel processing)
One Mapper per block (usually)
Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks
{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,
Containers define how many can run at once. }
Question: What happens if one DataNode goes down?
Answer:
Scenario
You have 5 DataNodes
Replication factor = 3
One DataNode crashes during job execution
NameNode detects failure
DataNodes send heartbeat every ~3 seconds
If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD
HDFS handles data automatically
Blocks on the dead DataNode are now under-replicated
NameNode schedules re-replication:
Copies missing blocks from healthy nodes
Restores replication factor
👉 No data loss
Running Map tasks on that node
Any mapper running on the failed node fails
YARN:
Reschedules the same map task
Runs it on another node that has a replica of the block
👉 Only failed tasks rerun, not the whole job
Reduce tasks
Reduce tasks are not tied to data locality
They continue normally unless the node running the reducer fails
Question: How to calculate optimal HDFS block size?
Answer:
Default block size
128 MB (modern clusters)
64 MB (older)
Balance between:
Parallelism
Task overhead
Disk & network efficiency
Use large blocks when:
Large files (GBs/TBs)
Sequential reads
Batch analytics
Use small blocks when:
Many small files
Low latency needed
Practical formula (industry-used)
Aim for 1–2 minutes per mapper
For your 120 GB file
We can provide custom value in $HADOOP_HOME/hadoop*.xml file:
<property>
<name>dfs.blocksize</name>
<value>268435456</value> <!-- 256 MB -->
</property>
🐘 Hadoop Architecture (Big Picture)
Hadoop has 3 core layers:
1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.
Architecture Components
NameNode (Master)
What it does
Stores metadata only (in memory):
File names
Block IDs
Block locations
Permissions
Decides:
Where blocks go
From where to read data
What it does NOT do
❌ Store actual data
❌ Handle client reads/writes directly
DataNode (Worker)
What it does
Stores actual data blocks
Sends heartbeats to NameNode
Sends block reports
On failure - NameNode re-replicates blocks
Secondary NameNode (Checkpoint Node) - NOT a backup NameNode
Purpose Periodically:
Merges fsimage + edits
Creates checkpoint
Reduces NameNode recovery time. Cannot take over automatically.
FSImage in HDFS:
1️⃣ What is FSImage?
FSImage is a persistent snapshot of the HDFS namespace metadata stored on disk.
It contains:
Directory structure
File names
Permissions
Block mapping (file → block IDs)
It does NOT contain actual data blocks.
2️⃣ Why FSImage Is Needed
NameNode metadata is stored in memory for fast access.
FSImage ensures:
Metadata survives NameNode restart
Faster recovery after failure
3️⃣ What FSImage Contains
✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files
❌ Block locations (DataNode info)
❌ Actual data blocks
📌 Block locations are provided by DataNodes via heartbeats.
Standby NameNode (HA)
Used in High Availability setup
✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination
What is ZooKeeper?
Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:
Configuration
Naming
Synchronization
Leader election
Failure detection
📌 It does NOT store application data
📌 It stores small metadata only
MapReduce (Built in 2004) - Indexing the webpages is equivalent to sorting of webpages.
MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.
MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.
Why MapReduce:
Huge amounts of data were stored in single servers prior to 2004.
The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.
With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.
MapReduce Word Count Problem:
Input
Splitting
Mapping
Shuffling
Reducing
Map Execution Phase
MapPhase
Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.
Partition Phase
Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.
Shuffle Phase
During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.
Sort Phase
The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.
Reduce Phase
The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.
Execution of MapReduce:
GFS (Google File System) / HDFS (Hadoop Distributed File System):
These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS).
By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.
Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.
For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB.
Total blocks = 100 GB / 1 GB = 100 blocks
HDFS does not assign blocks per node upfront.
Blocks are placed individually, not node-wise.
Some nodes might have 8 blocks, some 12, some 15, etc.
For replication factor = 5:
1 replica on local rack
2 replicas on a different rack
Remaining replicas distributed to balance load
If a node fails:
Tasks running on that node are rescheduled
Data is read from replica blocks on other nodes
No block is “lost” logically
NameNode detects under-replicated blocks and re-replicates them.
Question: What Shuffle Actually Does?
Answer: Shuffle is responsible for:
Partitioning – deciding which reducer gets which keys
Sorting – sorting intermediate keys
Grouping – grouping all values for the same key
Mapper Output:
(word, 1)
(word, 1)
(hadoop, 1)
(word, 1)
After Shuffle (Reducer Input):
(word, [1, 1, 1])
(hadoop, [1])
👉 All values of the same key are sent to the same reducer.
Question : What is a Reducer in Hadoop MapReduce?
Answer:
A reducer takes a key and all its associated values and combines them to produce final output records.
A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.
YARN:Yet Another Resource Negotiator
1️⃣ What is YARN?
YARN is Hadoop’s cluster resource management and job scheduling layer.
📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez
2️⃣ Why YARN Was Introduced
Problem before YARN (Hadoop 1.x)
MapReduce JobTracker:
Resource management
Job scheduling
Job monitoring
Single point of failure
Scalability limits
Solution - YARN separates resource management from processing
3️⃣ Core Components of YARN
🔹 1. ResourceManager (RM)
Global master
Allocates cluster resources
Does NOT execute tasks
Sub-components:
Scheduler (FIFO / Capacity / Fair) - 3 types of schedulers - Use for allocating the resources
FIFO and Fair are Spark job scheduling modes.
Capacity Scheduler is a YARN cluster scheduler, not a Spark scheduler.
✅ Scheduler
Decides how many containers
Decides which NodeManager
Decides resource size (CPU, memory)
Scheduler only allocates containers logically
It does NOT create containers physically
ApplicationsManager
🔹 2. NodeManager (NM)
Runs on every node
Manages containers
Monitors resource usage
Reports to ResourceManager
✅ NodeManager
Receives allocation info from RM
Creates the container
Launches the process
Monitors resource usage
Kills container if limits exceeded
Sends heartbeat to RM
Cleans up after completion
📌 Container = Linux process started by NodeManager
🔹 3. ApplicationMaster (AM)
One per application
Negotiates resources with RM
Manages application lifecycle
📌 Mandatory for every YARN app
🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.
Resource unit
Includes:
Memory
CPU
Disk
Network
📌 Tasks run inside containers
🔄 End-to-End Flow (Very Important)
ApplicationMaster requests containers
ResourceManager Scheduler allocates containers
NodeManager creates the container
NodeManager runs & monitors the container
NodeManager releases resources after completion
4️⃣ YARN Execution Flow (Step-by-Step)
Example: MapReduce Job
Client submits job to RM
RM launches ApplicationMaster
AM registers with RM
AM requests containers
RM allocates containers
NMs launch containers
Tasks execute
AM reports status
Job completes, resources released
5️⃣ YARN Scheduling Policies
6️⃣ YARN High Availability
Multiple ResourceManagers
ZooKeeper for:
Leader election
Failover
7️⃣ What YARN Does NOT Do ❌
❌ Store data
❌ Execute tasks itself
❌ Replace HDFS
8️⃣ YARN vs MapReduce
🧾 Summary
YARN separates resource management from data processing.
ResourceManager allocates resources but does not execute tasks.
Every YARN application has one ApplicationMaster.
Containers are the execution units in YARN.
YARN = cluster resource manager
Enables multi-engine Hadoop
Improves scalability & reliability
Backbone of modern Hadoop
6️⃣ YARN High Availability
Multiple ResourceManagers
ZooKeeper for:
Leader election
Failover
7️⃣ What YARN Does NOT Do ❌
❌ Store data
❌ Execute tasks itself
❌ Replace HDFS
8️⃣ YARN vs MapReduce
🧾 Summary
YARN separates resource management from data processing.
ResourceManager allocates resources but does not execute tasks.
Every YARN application has one ApplicationMaster.
Containers are the execution units in YARN.
YARN = cluster resource manager
Enables multi-engine Hadoop
Improves scalability & reliability
Backbone of modern Hadoop
No comments:
Post a Comment