HDFS (Hadoop Distributed File System) is the distributed storage layer of Hadoop. Its main purpose is to store very large files across multiple machines while providing scalability, fault tolerance, and high-throughput data access.
The easiest way to understand HDFS is through four concepts:
NameNode → DataNodes → Blocks → Replication
There are two fundamentally different types of information:
NameNode stores metadata
DataNodes store actual file data
This separation is the foundation of HDFS.
2. NameNode
The NameNode is the master metadata manager of HDFS.
It does not normally store the actual contents of your files.
File: sales.csv
File size: 500 MB
Replication: 3
Blocks: Block_01
Block_02
Block_03
Block_01 → DN1, DN3, DN5
Block_02 → DN2, DN4, DN6
Block_03 → DN1, DN2, DN5
What does the NameNode manage?
- File and directory hierarchy
- File permissions
- File → block mapping
- Block → DataNode mapping
- Replication information
- Namespace metadata
- DataNode health information
3. DataNode
A DataNode stores the actual HDFS blocks.
Suppose you upload: customer_data.csv
HDFS divides the file into blocks and distributes those blocks across DataNodes.
DataNodes are responsible for:
- Storing blocks
- Reading blocks
- Writing blocks
- Creating/deleting blocks
- Replicating blocks
- Sending heartbeats to NameNode
- Sending block reports to NameNode
4. What is an HDFS Block?
HDFS does not necessarily store an entire large file as one physical object.
Instead, it divides the file into blocks.
For example, imagine a simplified block size of 128 MB.
A 400 MB file could become: 400 MB file
┌──────────────┬──────────────┬─────────┐
│ Block 1 │ Block 2 │ Block 3 │ Block 4
│ 128 MB │ 128 MB │ 128 MB │ 16 MB └──────────────┴──────────────┴─────────┘
So:
File size = 400 MB
Block size = 128 MB
Number of blocks:
400 / 128 → 4 blocks
The last block contains only the remaining data.
Note: HDFS block size is configurable; 128 MB is a common example, not a universal fixed value.
5. Why does HDFS use blocks?
There are several important reasons.
Scalability
Large files can be distributed across many machines.
Parallel processing
Different blocks can be processed simultaneously.
Block 1 ──→ Machine 1 ──┐ Block 2 ──→ Machine 2 ──┼──→ Processing Block 3 ──→ Machine 3 ──┤ Block 4 ──→ Machine 4 ──┘
Fault tolerance
If one machine fails, replicated copies can be used.
Efficient distributed processing
Frameworks such as MapReduce and Spark can process blocks in parallel.
6. Replication
This is one of the most important HDFS concepts.
Suppose:
Block A
has a replication factor of 3.
HDFS maintains three copies:
Block A │ ┌───────┼───────┐ ▼ ▼ ▼ DN1 DN2 DN3 Copy 1 Copy 2 Copy 3
Now suppose DN2 fails:
DN1 DN2 DN3 ✓ ✗ ✓ Copy 1 FAILED Copy 3
HDFS still has two copies.
The NameNode detects that the replication level has fallen below the configured factor and schedules re-replication.
DN1 DN2 DN3 ✓ ✗ ✓ │ │ └──────────┬──────────────┘ │ Create another copy │ ▼ DN4
This is how HDFS provides fault tolerance.
7. How a File Is Written to HDFS
Let's walk through the process.
Suppose the client wants to upload:
sales.csv
Step 1 — Client contacts NameNode
Client │ │ "I want to write sales.csv" ▼ NameNode
The NameNode checks the filesystem and determines where blocks should be placed.
Step 2 — NameNode returns DataNodes
For example:
Block 1 → DN1, DN2, DN3 Block 2 → DN2, DN3, DN4
The NameNode gives the client the required metadata/location information.
Step 3 — Client writes directly to DataNodes
The client does not send the entire file through the NameNode.
Instead:
NameNode │ block locations │ ▼ Client ───────→ DN1 │ ▼ DN2 │ ▼ DN3
The actual data travels between the client and DataNodes.
This is a very important architectural distinction.
8. HDFS Write Pipeline
Replication commonly happens through a pipeline.
Suppose replication factor = 3:
Client │ │ Block data ▼ DN1 ─────────→ DN2 ─────────→ DN3 │ │ │ Copy 1 Copy 2 Copy 3
The client sends the block to DN1.
DN1 forwards it to DN2.
DN2 forwards it to DN3.
Each DataNode stores a copy.
The acknowledgements then travel back:
Client ▲ │ ACK │ DN1 ▲ │ ACK DN2 ▲ │ ACK DN3
This allows HDFS to maintain replicated copies while writing.
9. What happens when a DataNode fails?
This is a very common interview question.
Suppose:
Block A DN1 ✓ DN2 ✗ DN3 ✓
The NameNode detects the failure through heartbeats.
DN1 ── heartbeat ──→ NameNode DN2 ── X DN3 ── heartbeat ──→ NameNode
After the failed DataNode is recognized, the NameNode identifies blocks that have insufficient replication.
It then instructs healthy DataNodes to create additional replicas.
Before: DN1 → Block A DN2 → Block A ❌ DN3 → Block A After re-replication: DN1 → Block A DN3 → Block A DN4 → Block A
The system has restored the desired replication level.
10. NameNode vs DataNode
| Feature | NameNode | DataNode |
|---|---|---|
| Primary responsibility | Metadata management | Data storage |
| Stores actual file blocks | No | Yes |
| Maintains namespace | Yes | No |
| Tracks block locations | Yes | Reports them |
| Stores file metadata | Yes | No |
| Sends heartbeats | No | Yes |
| Sends block reports | No | Yes |
| Handles client metadata requests | Yes | No |
| Handles actual data I/O | Primarily coordinates | Yes |
Easy way to remember
NameNode = "Where is the data?"
DataNode = "Here is the data."
11. Heartbeat and Block Report
DataNodes continuously communicate their health/status to the NameNode.
Heartbeat
DataNode -> "I'm alive"→ NameNode
The NameNode uses this to determine whether a DataNode is responsive.
Block Report
DataNodes also report the blocks they currently store.
DN1 → NameNode: I have: Block A Block B Block D Block F
The NameNode uses this information to maintain an accurate view of the distributed filesystem.
12. Rack Awareness
HDFS doesn't blindly place replicas on machines.
It can consider rack topology.
Imagine:
Rack 1 Rack 2 DN1 DN4 DN2 DN5 DN3 DN6
Instead of putting all replicas inside the same rack:
Block A DN1 DN2 DN3
HDFS can distribute replicas across racks.
Block A DN1 ── Rack 1 DN2 ── Rack 1 DN5 ── Rack 2
Why?
Because a rack failure could potentially make multiple machines unavailable simultaneously.
Rack-aware placement therefore improves fault tolerance.
13. The Complete Picture
Put everything together:
CLIENT │ Metadata request │ ▼ ┌────────────┐ │ NameNode │ │ │ │ Metadata │ │ File→Block │ │ Block→DN │ └─────┬──────┘ │ Block locations │ ┌─────────────┼─────────────┐ ▼ ▼ ▼ ┌────────┐ ┌────────┐ ┌────────┐ │ DataNode│ │DataNode│ │DataNode│ │ DN1 │ │ DN2 │ │DN3 │ ├────────┤ ├────────┤ ├────────┤ │ Block A │ │ Block A │ │ Block A │ │ Block B │ │ Block C │ │ Block D │ └────────┘ └────────┘ └────────┘ │ │ │ └─────────────┼─────────────┘ │ Replication
The mental model
Think of HDFS as a distributed warehouse:
- NameNode = warehouse catalog/manager
- DataNodes = storage rooms
- Blocks = boxes
- Replication = duplicate boxes stored in different rooms
- Heartbeat = "I'm alive" message
- Block report = inventory report
- Rack awareness = keeping copies in different buildings/sections
The most important architectural principle is:
The NameNode manages metadata and coordinates the filesystem, while DataNodes store the actual blocks. HDFS divides large files into blocks and replicates those blocks across DataNodes to achieve scalable, fault-tolerant distributed storage.
No comments:
Post a Comment