What are Hadoop's core components and architecture?
Hadoop is an open-source framework for storing and processing very large datasets across a cluster of commodity computers. It provides distributed, fault-tolerant storage and batch processing with data-local computation, at the cost of operational complexity and relatively high latency compared with specialized cloud storage, Spark, or cloud data warehouses.
Read full answerHide full answer
Applies toHadoop 3.x
Core Architecture
Hadoop commonly combines three core components:
| Component | Responsibility | Important processes | Trade-off |
|---|---|---|---|
| HDFS | Distributed storage for large files | NameNode, DataNode | High throughput and fault tolerance, but metadata and small-file management add complexity |
| YARN | Resource management and scheduling | ResourceManager, NodeManager, ApplicationMaster | Flexible cluster utilization, but requires careful capacity planning |
| MapReduce | Distributed batch processing | Map task, shuffle, Reduce task | Mature and fault-tolerant, but slower than in-memory engines for iterative workloads |
HDFS: Distributed Storage
HDFS divides files into large blocks and stores replicas of those blocks across different nodes.
-
NameNode: Maintains the filesystem namespace, file-to-block mappings, and block-replica information received from DataNodes. It does not normally store user data itself.
-
DataNode: Stores actual data blocks on local disks and performs block creation, replication, deletion, and recovery.
-
Default block size: 128 MB in Hadoop 3.x.
-
Default replication factor: 3, meaning each block is normally stored on three DataNodes.
-
Data locality: MapReduce tasks are scheduled near the DataNodes holding their input, reducing network transfers.
-
Fault tolerance: If a replica is lost, HDFS re-replicates an under-replicated block on another DataNode.
Hadoop 3.x supports erasure coding, which can store redundant parity information instead of maintaining three complete replicas. This reduces storage overhead for suitable data, although reconstruction may require more CPU and network activity. Replication remains useful for frequently accessed or latency-sensitive data.
NameNode high availability uses an active NameNode and one or more standby NameNodes. JournalNodes maintain the shared edit log, while ZooKeeper-based automatic failover coordinates which NameNode becomes active.
YARN and MapReduce
YARN separates cluster resource management from processing-framework execution:
-
ResourceManager: Allocates cluster resources and schedules applications globally.
-
NodeManager: Manages resources and tasks on an individual worker node.
-
ApplicationMaster: Coordinates one application and requests resources from the ResourceManager.
-
Container: An allocation of resources, such as CPU and memory, in which a task executes.
A MapReduce job follows these steps:
-
Map: Each map task reads its input split and emits key-value pairs.
-
Combiner: Optionally performs local aggregation to reduce shuffled data. It is an optimization and must preserve the final result.
-
Partitioning: A partitioner assigns each map-output record to a reducer.
-
Shuffle, sort, and merge: Map output is sorted by key, spilled to disk when necessary, and transferred to reducers; corresponding partitions are merged during the shuffle.
-
Reduce: Each reducer processes its partition and writes the final output.
A task is re-executed if its worker fails, while HDFS preserves the input and intermediate map output needed for recovery.
Ecosystem and Current Role
-
Hive: SQL-based analysis and data warehousing on Hadoop.
-
HBase: Distributed, wide-column NoSQL database commonly used for random reads and writes.
-
Spark: Faster in-memory processing engine often running on YARN and reading HDFS.
-
Pig: Legacy data-flow language that has largely been superseded by Spark and other tools.
-
Sqoop: Retired Apache project formerly used for bulk transfers between relational databases and Hadoop.
-
Oozie: Workflow scheduler for Hadoop jobs.
-
ZooKeeper: Coordination service for distributed synchronization, leader election, and failover.
-
Spark SQL: SQL analytics engine frequently used alongside or instead of Hive.
Many organizations now use cloud object stores such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage together with Spark or cloud warehouses. Nevertheless, HDFS and YARN concepts remain important because they illustrate distributed storage, data locality, scheduling, replication, and fault tolerance.
Common Pitfalls
-
Confusing NameNode and DataNode roles: The NameNode manages metadata; DataNodes hold user data blocks.
-
Ignoring the small-files problem: Many small files create excessive metadata and scheduling overhead. Combine files when practical.
-
Assuming replication is always best: Erasure coding may be more efficient for cold data, while replication usually provides faster recovery.
-
Overlooking data locality: Poor scheduling can move data across the network and sharply reduce performance.
-
Treating Hadoop as low latency: MapReduce is optimized for large batch workloads, not millisecond responses.
-
Interview follow-ups: Expect questions about block replication, NameNode high availability, YARN scheduling, combiners versus reducers, data locality, and differences between HDFS, Hive, HBase, and Spark.