🔷 The 5 V’s of Big Data
Volume – How much data -> Refers to the huge amount of data generated and stored.
Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.
Variety – Different data types -> Refers to the different types and formats of data.
Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.
Value – Business usefulness -> Refers to the business insight or benefit extracted from data.
🔷Why RDBMS Fails
Vertical scaling limit
Expensive hardware
Single point of failure
🔷How Big Data Systems Help
Distributed Infrastructure
A distributed system is a collection of independent computers that work together as a single system.
CAP Theorem
In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.
🔹 The 3 CAP Properties
Consistency (C)
All nodes see the same data at the same time.
📌 After a write, every read gets the latest value.
Availability (A)
Every request receives a response, even if it may not be the latest data.
📌 System is always responsive.
Network Partition Tolerance (P)
The system continues to operate despite network failures between nodes.
📌 Network partitions are unavoidable in distributed systems.
Distributed Storage
Communication
Distributed Computation
Implementation:
Remote Procedure Calls (RPC)
Threads
Concurrency Control
Performance:
Scalability (2 * computation power -> 2* throughput)
Computational Speed - reduced timing for programme execution
Fault Tolerance:
Availability
Recoverability
Replication
Consistency:
Put Operation (Key, Value)
Get Operation (Key, value)
🔹 Rack
A rack is a physical group of nodes connected to the same network switch.
Hadoop
Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.
Components:
HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.
YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.
MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.
Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.
Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
“Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
Example
Replication factor = 3
Each block is stored 3 times on 3 different DataNodes
So if you have a 128 MB block:
Block 1 (128 MB) → DN1
Block 1 (128 MB) → DN2
Block 1 (128 MB) → DN3
1️⃣ Fault tolerance - If one DataNode fails → data still available
2️⃣ High availability - Client can read from nearest replica
3️⃣ Better read performance - Parallel reads from different nodes
🧠 How replication works internally
If a DataNode fails:
DataNodes send heartbeats to NameNode:
HDFS must always maintain 3 healthy copies of every block (depends on replication factor)
🔹Self-Healing Mechanism (CORE HDFS FEATURE)
HDFS is designed to: Detect → Decide → Replicate → Recover
HDFS’s standard rack awareness policy dictates two main rules:
No single rack should hold more than two replicas of the same block.
Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.
NameNode detects missing block replica
Marks block as under-replicated
Selects:
Steps:
NameNode sends metadata instruction (RPC) Remote Procedure Calls:
DN1 replicate block B1 to DN4
DN1 opens a TCP connection to DN4
Data is transferred using DataTransferProtocol (using TCP)
Checksums are verified during transfer
DN4 confirms successful write
NameNode updates metadata
> Actual data never goes through the NameNode
Rebalancer is a manual administrative tool used to:
🔸 When Is Rebalancer Used?
New DataNodes added
Some nodes are full
Uneven disk utilization
🔸 Who Triggers It?
❌ Not automatic
✅ Admin manually runs it
🔸 Purpose
Balance storage usage
Improve performance
Prevent hotspots
Question: What exactly happens when a SINGLE file is processed using Hadoop?
Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.
Answer:
Number of blocks:
122,880 MB / 128 MB ≈ 960 blocks
The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies
Job submission (MapReduce)
Client submit jobs to YARN ResourceManager.
ResourceManager asks NameNode: Where are the blocks stored?
Jobs are divided into tasks.
Map phase (parallel processing)
One Mapper per block (usually)
Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks
{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,
Containers define how many can run at once. }
Question: What happens if one DataNode goes down?
Answer:
Scenario
NameNode detects failure
DataNodes send heartbeat every ~3 seconds
If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD
HDFS handles data automatically
👉 No data loss
Running Map tasks on that node
👉 Only failed tasks rerun, not the whole job
Reduce tasks
Question: How to calculate optimal HDFS block size?
Answer:
Default block size
128 MB (modern clusters)
64 MB (older)
Balance between:
Use large blocks when:
Large files (GBs/TBs)
Sequential reads
Batch analytics
Use small blocks when:
Many small files
Low latency needed
Practical formula (industry-used)
For your 120 GB file
Block size | No of Blocks | Mapper waves (5 nodes) |
128 MB | ~960 | Too many |
256 MB | ~480 | Better |
512 MB | ~240 | Best |
1 GB | ~120 | Good if memory allows |
We can provide custom value in $HADOOP_HOME/hadoop*.xml file:
<property>
<name>dfs.blocksize</name>
<value>268435456</value> <!-- 256 MB -->
</property>
🐘 Hadoop Architecture (Big Picture)
1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.
Architecture Components
What it does
What it does NOT do
❌ Store actual data
❌ Handle client reads/writes directly
What it does
On failure - NameNode re-replicates blocks
Secondary NameNode (Checkpoint Node) - NOT a backup NameNode
Merges fsimage + edits
Creates checkpoint
Reduces NameNode recovery time. Cannot take over automatically.
1️⃣ What is FSImage?
FSImage is a persistent snapshot of the HDFS namespace metadata stored on disk.
It contains:
It does NOT contain actual data blocks.
2️⃣Why FSImage Is Needed
NameNode metadata is stored in memory for fast access.
FSImage ensures:
3️⃣ What FSImage Contains
✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files
❌ Block locations (DataNode info)
❌ Actual data blocks
📌 Block locations are provided by DataNodes via heartbeats.
Used in High Availability setup
✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination
Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:
Configuration
Naming
Synchronization
Leader election
Failure detection
📌 It does NOT store application data
📌 It stores small metadata only
MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.
MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.
Why MapReduce:
Huge amounts of data were stored in single servers prior to 2004.
The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.
With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.
MapReduce Word Count Problem:
Input
Splitting
Mapping
Shuffling
Reducing
Map Execution Phase
Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.
Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.
During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.
The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.
The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.
Execution of MapReduce:
These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS).
By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.
Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.
For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB.
Total blocks = 100 GB / 1 GB = 100 blocks
Some nodes might have 8 blocks, some 12, some 15, etc.
For replication factor = 5:
If a node fails:
Tasks running on that node are rescheduled
Data is read from replica blocks on other nodes
No block is “lost” logically
NameNode detects under-replicated blocks and re-replicates them.
Question: What Shuffle Actually Does?
Answer: Shuffle is responsible for:
Partitioning – deciding which reducer gets which keys
Sorting – sorting intermediate keys
Grouping – grouping all values for the same key
Mapper Output:
(word, 1)
(word, 1)
(hadoop, 1)
(word, 1)
After Shuffle (Reducer Input):
(word, [1, 1, 1])
(hadoop, [1])
👉 All values of the same key are sent to the same reducer.
Question : What is a Reducer in Hadoop MapReduce?
Answer: A reducer takes a key and all its associated values and combines them to produce final output records.
A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.
YARN:Yet Another Resource Negotiator
1️⃣ What is YARN?
YARN is Hadoop’s cluster resource management and job scheduling layer.
📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez
2️⃣ Why YARN Was Introduced
Problem before YARN (Hadoop 1.x)
MapReduce JobTracker:
Resource management
Job scheduling
Job monitoring
Single point of failure
Scalability limits
Solution - YARN separates resource management from processing
3️⃣ Core Components of YARN
🔹 1. ResourceManager (RM)
Sub-components:
Feature | FIFO (Spark) | Fair (Spark) | Capacity (YARN) |
Scope | Single Spark application | Single Spark application | Entire cluster |
Scheduling | First submitted job first | Resources shared fairly | Queue-based resource allocation |
Multi-user support | Poor | Good | Excellent |
Starvation | Possible | Much less likely | Prevented through queue guarantees |
Best for | Development, single workload | Shared Spark environments | Enterprise clusters |
✅ Scheduler
Decides how many containers
Decides which NodeManager
Decides resource size (CPU, memory)
Scheduler only allocates containers logically
It does NOT create containers physically
ApplicationsManager
🔹 2. NodeManager (NM)
✅ NodeManager
📌 Container = Linux process started by NodeManager
🔹 3. ApplicationMaster (AM)
📌 Mandatory for every YARN app
🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.
📌 Tasks run inside containers
🔄 End-to-End Flow (Very Important)
ApplicationMaster requests containers
ResourceManager Scheduler allocates containers
NodeManager creates the container
NodeManager runs & monitors the container
NodeManager releases resources after completion
4️⃣ YARN Execution Flow (Step-by-Step)
Example: MapReduce Job
Client submits job to RM
RM launches ApplicationMaster
AM registers with RM
AM requests containers
RM allocates containers
NMs launch containers
Tasks execute
AM reports status
Job completes, resources released
5️⃣ YARN Scheduling Policies
Big Data
Big Data refers to datasets that are too large, fast, or complex to be processed using traditional systems.
🔷 The 5 V’s of Big Data
Volume – How much data -> Refers to the huge amount of data generated and stored.
Velocity – How fast data arrives -> Refers to the speed at which data is generated, ingested, and processed.
Variety – Different data types -> Refers to the different types and formats of data.
Veracity – Data quality -> Refers to the quality, accuracy, and trustworthiness of data.
Value – Business usefulness -> Refers to the business insight or benefit extracted from data.
🔷Why RDBMS Fails
Vertical scaling limit
Expensive hardware
Single point of failure
🔷How Big Data Systems Help
Distributed Infrastructure
A distributed system is a collection of independent computers that work together as a single system.
CAP Theorem
In a distributed system, it is impossible to simultaneously guarantee all three:
Consistency, Availability, and Partition Tolerance.
🔹 The 3 CAP Properties
Consistency (C)
All nodes see the same data at the same time.
📌 After a write, every read gets the latest value.
Availability (A)
Every request receives a response, even if it may not be the latest data.
📌 System is always responsive.
Network Partition Tolerance (P)
The system continues to operate despite network failures between nodes.
📌 Network partitions are unavoidable in distributed systems.
Distributed Storage
Communication
Distributed Computation
Implementation:
Remote Procedure Calls (RPC)
Threads
Concurrency Control
Performance:
Scalability (2 * computation power -> 2* throughput)
Computational Speed - reduced timing for programme execution
Fault Tolerance:
Availability
Recoverability
Replication
Consistency:
Put Operation (Key, Value)
Get Operation (Key, value)
🔹 Rack
A rack is a physical group of nodes connected to the same network switch.
Hadoop
Hadoop is an open source software framework that is used for storing and processing large amounts of data in a distributed computing environment, it is designed to handle big data and is based on mapReduce programming model, which allows for the parallel processing of large datasets. Its framework is based on Java programming with some native code in C and shell scripting.
Components:
HDFS (Hadoop Distributed File System) (storage): A distributed storage system that splits files into blocks and distributes them across a cluster. HDFS breaks big files into blocks and spreads them across a cluster of machines with replication factor.
YARN (resource management): (Yet Another Resource Negotiator): Manages resources and schedules jobs in the cluster.
MapReduce (processing) : A programming model for processing data across parallel nodes. MapReduce is the computing engine that processes data in a distributed manner. It splits large tasks into smaller chunks (map) and then merges the results (reduce), allowing Hadoop to quickly process massive datasets.
Hadoop Common (utilities): A set of shared libraries and utilities that support other modules.
Replication Factor is a core HDFS concept. Number of copies of each HDFS block stored on different DataNodes.
“Replication factor defines how many copies of each HDFS block are stored across different DataNodes to ensure fault tolerance and availability.”
Example
Replication factor = 3
Each block is stored 3 times on 3 different DataNodes
So if you have a 128 MB block:
Block 1 (128 MB) → DN1
Block 1 (128 MB) → DN2
Block 1 (128 MB) → DN3
1️⃣ Fault tolerance - If one DataNode fails → data still available
2️⃣ High availability - Client can read from nearest replica
3️⃣ Better read performance - Parallel reads from different nodes
🧠 How replication works internally
If a DataNode fails:
DataNodes send heartbeats to NameNode:
HDFS must always maintain 3 healthy copies of every block (depends on replication factor)
🔹Self-Healing Mechanism (CORE HDFS FEATURE)
HDFS is designed to: Detect → Decide → Replicate → Recover
HDFS’s standard rack awareness policy dictates two main rules:
No single rack should hold more than two replicas of the same block.
Replicas must be spread across at least two different racks to prevent data loss if a whole rack fails.
NameNode detects missing block replica
Marks block as under-replicated
Selects:
Steps:
NameNode sends metadata instruction (RPC) Remote Procedure Calls:
DN1 replicate block B1 to DN4
DN1 opens a TCP connection to DN4
Data is transferred using DataTransferProtocol (using TCP)
Checksums are verified during transfer
DN4 confirms successful write
NameNode updates metadata
> Actual data never goes through the NameNode
Rebalancer is a manual administrative tool used to:
🔸 When Is Rebalancer Used?
New DataNodes added
Some nodes are full
Uneven disk utilization
🔸 Who Triggers It?
❌ Not automatic
✅ Admin manually runs it
🔸 Purpose
Balance storage usage
Improve performance
Prevent hotspots
Question: What exactly happens when a SINGLE file is processed using Hadoop?
Note: The default HDFS block size is 128 MB in Hadoop 2.x and 3.x, increased from 64 MB in Hadoop 1.x to optimize performance and reduce NameNode overhead for large no of datasets.
Answer:
Number of blocks:
122,880 MB / 128 MB ≈ 960 blocks
The file is split into ~960 blocks
Each block is stored on different DataNodes
With replication factor = 3, each block has 3 copies
Job submission (MapReduce)
Client submit jobs to YARN ResourceManager.
ResourceManager asks NameNode: Where are the blocks stored?
Jobs are divided into tasks.
Map phase (parallel processing)
One Mapper per block (usually)
Each 128 MB block → 1 Map task
For ~960 blocks → ~960 map tasks
{ Key rule (very important): Number of parallel mappers = available YARN containers, NOT number of HDFS blocks.
Blocks define how many map tasks exist,
Containers define how many can run at once. }
Question: What happens if one DataNode goes down?
Answer:
Scenario
NameNode detects failure
DataNodes send heartbeat every ~3 seconds
If no heartbeat for ~10 minutes (configurable), NameNode marks node as DEAD
HDFS handles data automatically
👉 No data loss
Running Map tasks on that node
👉 Only failed tasks rerun, not the whole job
Reduce tasks
Question: How to calculate optimal HDFS block size?
Answer:
Default block size
128 MB (modern clusters)
64 MB (older)
Balance between:
Use large blocks when:
Large files (GBs/TBs)
Sequential reads
Batch analytics
Use small blocks when:
Many small files
Low latency needed
Practical formula (industry-used)
For your 120 GB file
Block size | No of Blocks | Mapper waves (5 nodes) |
128 MB | ~960 | Too many |
256 MB | ~480 | Better |
512 MB | ~240 | Best |
1 GB | ~120 | Good if memory allows |
We can provide custom value in $HADOOP_HOME/hadoop*.xml file:
<property>
<name>dfs.blocksize</name>
<value>268435456</value> <!-- 256 MB -->
</property>
🐘 Hadoop Architecture (Big Picture)
1️⃣ HDFS – Storage
2️⃣ YARN – Resource Management
3️⃣ Processing Engines – MapReduce, Spark, Hive, etc.
Architecture Components
What it does
What it does NOT do
❌ Store actual data
❌ Handle client reads/writes directly
What it does
On failure - NameNode re-replicates blocks
Secondary NameNode (Checkpoint Node) - NOT a backup NameNode
Merges fsimage + edits
Creates checkpoint
Reduces NameNode recovery time. Cannot take over automatically.
1️⃣ What is FSImage?
FSImage is a persistent snapshot of the HDFS namespace metadata stored on disk.
It contains:
It does NOT contain actual data blocks.
2️⃣ Why FSImage Is Needed
NameNode metadata is stored in memory for fast access.
FSImage ensures:
3️⃣ What FSImage Contains
✔ File & directory hierarchy
✔ Ownership & permissions
✔ File replication factor
✔ Block IDs associated with files
❌ Block locations (DataNode info)
❌ Actual data blocks
📌 Block locations are provided by DataNodes via heartbeats.
Used in High Availability setup
✔ Automatically takes over if Active NameNode fails
✔ Uses ZooKeeper for coordination
Apache ZooKeeper is a centralized coordination service that helps distributed systems manage:
Configuration
Naming
Synchronization
Leader election
Failure detection
📌 It does NOT store application data
📌 It stores small metadata only
MapReduce is a programming model that simultaneously processes and analyzes huge data sets into separate clusters, while Map sorts the data, Reduce segregates it into logical clusters, thus removing the bad data and retaining the necessary information.
MapReduce was introduced by Google engineers Jeffrey Dean and Sanjay Ghemawat in 2004 to simplify processing massive datasets on large, distributed clusters.
Why MapReduce:
Huge amounts of data were stored in single servers prior to 2004.
The threat of data loss, challenge of data backup, and reduced scalability resulted in the issue snowballing into a crisis of sorts.
With MapReduce, Queries could run simultaneously on multiple servers, search results could be logically integrated, and data could be analyzed in real time.
MapReduce Word Count Problem:
Input
Splitting
Mapping
Shuffling
Reducing
Map Execution Phase
Reads assigned input split using InputFormat and RecordReader, converts input into key–value records, applies the map function to each record, and reports task completion to the ApplicationMaster.
Each mapper uses a partitioner (default: HashPartitioner) to determine which reducer receives each output key.
All identical keys are sent to the same partition, and the Number of Partitions = Number of Reducers.
During shuffle, each reducer fetches its corresponding partition of intermediate data from all mappers, transferring data over the network and preparing it for sorting and grouping.
The reducer merge-sorts the fetched mapper outputs by key, grouping all values associated with the same key together.
The reduce function is applied to each key and its grouped values, and the final output is written to HDFS via the OutputFormat.
Execution of MapReduce:
These file systems will automatically distribute the file in data chunks called BlocksBlocks (128 MB by default in HDFS; 64MB in GFS).
By default, HDFS follows a rack-aware placement policy where 1 replica is placed on the local rack, at least two replicas on a different rack, and remaining replicas are distributed to balance load.
Replication factor greater than 1 provides data durability and fault tolerance against node and rack failures.
For example if we have 10 nodes and replication factor is 5 and file size is 100 GB and we have provided the Default Block size is 1 GB.
Total blocks = 100 GB / 1 GB = 100 blocks
Some nodes might have 8 blocks, some 12, some 15, etc.
For replication factor = 5:
If a node fails:
Tasks running on that node are rescheduled
Data is read from replica blocks on other nodes
No block is “lost” logically
NameNode detects under-replicated blocks and re-replicates them.
Question: What Shuffle Actually Does?
Answer: Shuffle is responsible for:
Partitioning – deciding which reducer gets which keys
Sorting – sorting intermediate keys
Grouping – grouping all values for the same key
Mapper Output:
(word, 1)
(word, 1)
(hadoop, 1)
(word, 1)
After Shuffle (Reducer Input):
(word, [1, 1, 1])
(hadoop, [1])
👉 All values of the same key are sent to the same reducer.
Question : What is a Reducer in Hadoop MapReduce?
Answer:
A reducer takes a key and all its associated values and combines them to produce final output records.
A reducer is the component (task) in MapReduce that processes the grouped output produced after the shuffle and sort phase and generates the final result of a job.
YARN:Yet Another Resource Negotiator
1️⃣ What is YARN?
YARN is Hadoop’s cluster resource management and job scheduling layer.
📌 It does not process data itself
📌 It manages resources for applications like MapReduce, Spark, Hive, Tez
2️⃣ Why YARN Was Introduced
Problem before YARN (Hadoop 1.x)
MapReduce JobTracker:
Resource management
Job scheduling
Job monitoring
Single point of failure
Scalability limits
Solution - YARN separates resource management from processing
3️⃣ Core Components of YARN
🔹 1. ResourceManager (RM)
Sub-components:
Feature | FIFO (Spark) | Fair (Spark) | Capacity (YARN) |
Scope | Single Spark application | Single Spark application | Entire cluster |
Scheduling | First submitted job first | Resources shared fairly | Queue-based resource allocation |
Multi-user support | Poor | Good | Excellent |
Starvation | Possible | Much less likely | Prevented through queue guarantees |
Best for | Development, single workload | Shared Spark environments | Enterprise clusters |
✅ Scheduler
Decides how many containers
Decides which NodeManager
Decides resource size (CPU, memory)
Scheduler only allocates containers logically
It does NOT create containers physically
ApplicationsManager
🔹 2. NodeManager (NM)
✅ NodeManager
📌 Container = Linux process started by NodeManager
🔹 3. ApplicationMaster (AM)
📌 Mandatory for every YARN app
🔹 4. Container -> ResourceManager allocates containers, but NodeManager creates and manages them.
📌 Tasks run inside containers
🔄 End-to-End Flow (Very Important)
ApplicationMaster requests containers
ResourceManager Scheduler allocates containers
NodeManager creates the container
NodeManager runs & monitors the container
NodeManager releases resources after completion
4️⃣ YARN Execution Flow (Step-by-Step)
Example: MapReduce Job
Client submits job to RM
RM launches ApplicationMaster
AM registers with RM
AM requests containers
RM allocates containers
NMs launch containers
Tasks execute
AM reports status
Job completes, resources released
5️⃣ YARN Scheduling Policies
6️⃣ YARN High Availability
7️⃣ What YARN Does NOT Do ❌
❌ Store data
❌ Execute tasks itself
❌ Replace HDFS
8️⃣ YARN vs MapReduce
🧾 Summary
YARN separates resource management from data processing.
ResourceManager allocates resources but does not execute tasks.
Every YARN application has one ApplicationMaster.
Containers are the execution units in YARN.
YARN = cluster resource manager
Enables multi-engine Hadoop
Improves scalability & reliability
Backbone of modern Hadoop
6️⃣ YARN High Availability
7️⃣ What YARN Does NOT Do ❌
❌ Store data
❌ Execute tasks itself
❌ Replace HDFS
8️⃣ YARN vs MapReduce
🧾 Summary
YARN separates resource management from data processing.
ResourceManager allocates resources but does not execute tasks.
Every YARN application has one ApplicationMaster.
Containers are the execution units in YARN.
YARN = cluster resource manager
Enables multi-engine Hadoop
Improves scalability & reliability
Backbone of modern Hadoop
