Skip to content
Machine Learning & Data Science

Hadoop interview questions and answers

50 questions, 26 with code. About 4 h 55 min to read them all.

By TestLab · Updated

Plan for an interview
50 questions

Track what you know and what to revise.

Sign in to track progress

You can open a few full answers without signing in. Sign in to read all

1

What are Hadoop's core components and architecture?

Hadoop is an open-source framework for storing and processing very large datasets across a cluster of commodity computers. It provides distributed, fault-tolerant storage and batch processing with data-local computation, at the cost of operational complexity and relatively high latency compared with specialized cloud storage, Spark, or cloud data warehouses.

Read full answer

Applies toHadoop 3.x

Core Architecture

Hadoop commonly combines three core components:

Component Responsibility Important processes Trade-off
HDFS Distributed storage for large files NameNode, DataNode High throughput and fault tolerance, but metadata and small-file management add complexity
YARN Resource management and scheduling ResourceManager, NodeManager, ApplicationMaster Flexible cluster utilization, but requires careful capacity planning
MapReduce Distributed batch processing Map task, shuffle, Reduce task Mature and fault-tolerant, but slower than in-memory engines for iterative workloads

HDFS: Distributed Storage

HDFS divides files into large blocks and stores replicas of those blocks across different nodes.

  • NameNode: Maintains the filesystem namespace, file-to-block mappings, and block-replica information received from DataNodes. It does not normally store user data itself.

  • DataNode: Stores actual data blocks on local disks and performs block creation, replication, deletion, and recovery.

  • Default block size: 128 MB in Hadoop 3.x.

  • Default replication factor: 3, meaning each block is normally stored on three DataNodes.

  • Data locality: MapReduce tasks are scheduled near the DataNodes holding their input, reducing network transfers.

  • Fault tolerance: If a replica is lost, HDFS re-replicates an under-replicated block on another DataNode.

Hadoop 3.x supports erasure coding, which can store redundant parity information instead of maintaining three complete replicas. This reduces storage overhead for suitable data, although reconstruction may require more CPU and network activity. Replication remains useful for frequently accessed or latency-sensitive data.

NameNode high availability uses an active NameNode and one or more standby NameNodes. JournalNodes maintain the shared edit log, while ZooKeeper-based automatic failover coordinates which NameNode becomes active.

YARN and MapReduce

YARN separates cluster resource management from processing-framework execution:

  • ResourceManager: Allocates cluster resources and schedules applications globally.

  • NodeManager: Manages resources and tasks on an individual worker node.

  • ApplicationMaster: Coordinates one application and requests resources from the ResourceManager.

  • Container: An allocation of resources, such as CPU and memory, in which a task executes.

A MapReduce job follows these steps:

  1. Map: Each map task reads its input split and emits key-value pairs.

  2. Combiner: Optionally performs local aggregation to reduce shuffled data. It is an optimization and must preserve the final result.

  3. Partitioning: A partitioner assigns each map-output record to a reducer.

  4. Shuffle, sort, and merge: Map output is sorted by key, spilled to disk when necessary, and transferred to reducers; corresponding partitions are merged during the shuffle.

  5. Reduce: Each reducer processes its partition and writes the final output.

A task is re-executed if its worker fails, while HDFS preserves the input and intermediate map output needed for recovery.

Ecosystem and Current Role

  • Hive: SQL-based analysis and data warehousing on Hadoop.

  • HBase: Distributed, wide-column NoSQL database commonly used for random reads and writes.

  • Spark: Faster in-memory processing engine often running on YARN and reading HDFS.

  • Pig: Legacy data-flow language that has largely been superseded by Spark and other tools.

  • Sqoop: Retired Apache project formerly used for bulk transfers between relational databases and Hadoop.

  • Oozie: Workflow scheduler for Hadoop jobs.

  • ZooKeeper: Coordination service for distributed synchronization, leader election, and failover.

  • Spark SQL: SQL analytics engine frequently used alongside or instead of Hive.

Many organizations now use cloud object stores such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage together with Spark or cloud warehouses. Nevertheless, HDFS and YARN concepts remain important because they illustrate distributed storage, data locality, scheduling, replication, and fault tolerance.

Common Pitfalls

  • Confusing NameNode and DataNode roles: The NameNode manages metadata; DataNodes hold user data blocks.

  • Ignoring the small-files problem: Many small files create excessive metadata and scheduling overhead. Combine files when practical.

  • Assuming replication is always best: Erasure coding may be more efficient for cold data, while replication usually provides faster recovery.

  • Overlooking data locality: Poor scheduling can move data across the network and sharply reduce performance.

  • Treating Hadoop as low latency: MapReduce is optimized for large batch workloads, not millisecond responses.

  • Interview follow-ups: Expect questions about block replication, NameNode high availability, YARN scheduling, combiners versus reducers, data locality, and differences between HDFS, Hive, HBase, and Spark.

2

What is the architecture of HDFS in Hadoop?

Hadoop Distributed File System (HDFS) is a distributed, block-based file system designed to store large datasets across a cluster. It separates metadata management from block storage, providing fault-tolerant, high-throughput storage for data-intensive workloads. Its main trade-off is optimization for write-once/read-many and sequential processing rather than low-latency random updates.

Read full answer

Applies toApache Hadoop 3.x

Core Architecture

Component Responsibility Key characteristics
NameNode Manages the filesystem namespace Stores the directory hierarchy, permissions, and file-to-block mappings; it does not store user data
DataNode Stores and serves HDFS blocks Keeps replicas on local disks, sends heartbeats and block reports, and handles read/write requests
Secondary NameNode Produces namespace checkpoints in conventional deployments It is not a hot standby and does not provide automatic failover
JournalNode Maintains the shared edit log in an HA deployment A quorum of JournalNodes coordinates metadata edits among NameNodes
HDFS client Provides filesystem access Obtains block locations from the NameNode and transfers data directly between clients and DataNodes

Hadoop 3.x uses a default HDFS block size of 128 MB and replication factor of 3. A file is divided into blocks, with each replicated block normally placed on separate DataNodes. Without high availability, the active NameNode is a single point of failure.

Data Flow, Fault Tolerance, and High Availability

For a conventionally replicated write, the client asks the NameNode where the replicas should be placed. It then sends packets through a pipeline of DataNodes, with each DataNode storing a copy and forwarding the packet to the next. Packets contain checksums, allowing corruption to be detected.

For a read, the client asks the NameNode for block locations and contacts the DataNodes directly. MapReduce and other YARN applications can use data locality by scheduling computation close to the input blocks and reducing network transfers.

  • DataNode failure: The NameNode identifies under-replicated blocks and directs other DataNodes to restore the missing replicas.

  • Block corruption: Checksums identify unreadable blocks, which the NameNode re-replicates from a healthy replica.

  • Task failure: YARN and MapReduce can re-execute failed tasks, subject to the configured retry limits.

  • Erasure coding: Hadoop 3.x can encode data across multiple DataNodes with lower storage overhead than replication, at the cost of extra computation during reads and reconstruction.

  • NameNode high availability: An active NameNode and one or more standby NameNodes maintain state through JournalNodes. ZKFailoverController uses ZooKeeper coordination to trigger automatic failover.

MapReduce and the Hadoop Ecosystem

A MapReduce job follows these steps:

  1. Input split: The InputFormat computes splits, commonly aligned with HDFS block boundaries.

  2. Map: Each mapper processes a split and emits key-value pairs such as word -> 1.

  3. Combine: An optional combiner performs local partial aggregation to reduce shuffled data.

  4. Partition: A partitioner, usually hash-based, selects the reducer for each key.

  5. Shuffle and sort: Reducers fetch map output and merge sorted runs grouped by key.

  6. Reduce: Each reducer aggregates the values for its keys and writes the results to HDFS.

Tool How it works Strengths Weaknesses When to use / typical scale
YARN ResourceManager allocates resources, NodeManagers run containers, and an ApplicationMaster coordinates each job Shared cluster resource management Operational complexity Multi-tenant clusters with concurrent applications
MapReduce Processes data through map, shuffle, and reduce phases Reliable, fault-tolerant batch processing High latency and awkward iterative workflows Large historical batch ETL workloads
Hive Translates HiveQL into execution plans SQL-friendly warehouse analytics Not intended for low-latency row serving Large analytical databases and data warehouses
HBase Stores rows in distributed, wide-column tables Low-latency random key-based access Operational complexity and sensitive row-key design Large sparse tables requiring online lookups
Spark Executes DataFrame, SQL, and RDD workloads through a distributed engine Faster iterative, streaming, and in-memory processing Greater memory and tuning demands Medium-latency analytics and pipelines, often on YARN

Pig is a legacy data-flow language, Sqoop is a retired Apache project, Oozie manages workflows, and ZooKeeper provides distributed coordination. Spark often runs on YARN and reads HDFS, although many teams now use S3, GCS, ADLS, or managed cloud warehouses instead.

Common Pitfalls

  • Small files: Millions of tiny files consume excessive NameNode memory and increase task overhead; compact or archive them into larger files.

  • Treating the NameNode as a data server: It manages metadata; clients transfer block data directly from DataNodes.

  • Ignoring replication costs: Replication improves availability and read throughput but increases storage and network usage.

  • Ignoring data locality: Moving computation to the data is often substantially cheaper than moving large datasets across the network.

  • Ignoring failover readiness: Test NameNode failover, JournalNode recovery, and namespace restoration rather than assuming HA configuration is sufficient.

3

How does Hadoop execute MapReduce jobs?

MapReduce is Hadoop’s batch execution model: YARN schedules tasks, HDFS supplies replicated input and output, and failed task attempts are retried. Mappers emit key-value pairs; the framework partitions, sorts, and shuffles them so reducers aggregate each key. Its main trade-off is simple, fault-tolerant scale versus high latency and disk/network-heavy shuffles, making it less suitable than Spark for iterative or interactive workloads.

Read full answer

Applies toHadoop 3.x with Hadoop Streaming and Python 3.12 examples

How MapReduce Works

  1. Input splitting: InputFormat divides input into splits, usually aligned with HDFS blocks so computation can move to the data.

  2. Map phase: Each map task processes records and emits intermediate key-value pairs.

  3. Shuffle and sort: Map output is partitioned by key, spilled to local disk, sorted, merged, and transferred to the appropriate reducers.

  4. Reduce phase: Each reducer receives the values for its assigned keys and produces the final output.

  5. Output writing: OutputFormat writes results, normally as files in HDFS.

The default HashPartitioner uses (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks, ensuring that keys assigned to the same reducer are delivered together.

Architecture and Fault Tolerance

  • HDFS: The NameNode stores namespace and block metadata; DataNodes store the data in blocks. The Hadoop 3.x default block size is 128 MB and the default replication factor is 3.

  • YARN: The ResourceManager allocates resources, NodeManagers manage containers on workers, and the MapReduce ApplicationMaster coordinates task execution.

  • Data locality: Map tasks are preferentially scheduled on nodes holding their input blocks. This reduces source-data network traffic, although shuffled map output still moves across the network.

  • Failure recovery: HDFS reconstructs missing replicas, while the ApplicationMaster reschedules failed map or reduce task attempts. Speculative execution can duplicate a slow task.

  • High availability and storage efficiency: Hadoop 3.x supports multiple standby NameNodes. It also supports erasure-coding policies as an alternative to replication, reducing storage overhead at the cost of additional computation and reconstruction work.

Word Count Step by Step

For input containing hadoop map reduce and hadoop map:

  • The mapper emits each token with a count of 1.

  • An optional combiner performs a local mini-reduce, reducing network and disk traffic.

  • The partitioner assigns each word to one reducer.

  • The shuffle phase groups values by key and presents them in sorted order.

  • The reducer sums the values, producing (hadoop, 2), (map, 2), and (reduce, 1).

Combiners are safe only when their operation is associative and commutative and does not require the complete global dataset.

Code Example

Save these scripts as mapper.py and reducer.py. Ensure Python 3.12 is available as python3.12 on the worker nodes.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")
# reducer.py
import sys

current_word = None
total = 0

# Hadoop Streaming supplies reducer input sorted by key.
for line in sys.stdin:
    word, count_text = line.rstrip("\n").split("\t", 1)
    count = int(count_text)

    if current_word is None:
        current_word = word
        total = count
    elif word == current_word:
        total += count
    else:
        print(f"{current_word}\t{total}")
        current_word = word
        total = count

if current_word is not None:
    print(f"{current_word}\t{total}")

Create the input, upload it to HDFS, and run the Streaming job:

cat > input.txt <<'EOF'
hadoop map reduce
hadoop map
EOF

hdfs dfs -mkdir -p "/user/$USER/mapreduce-wordcount"
hdfs dfs -put -f input.txt "/user/$USER/mapreduce-wordcount/input.txt"
hdfs dfs -ls -h "/user/$USER/mapreduce-wordcount"

STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' | sort -V | tail -n 1)"

if [ -z "$STREAMING_JAR" ]; then
  echo "Hadoop Streaming JAR not found" >&2
  exit 1
fi

OUTPUT="/user/$USER/mapreduce-wordcount/output-$(date +%s)"

hadoop jar "$STREAMING_JAR" \
  -input "/user/$USER/mapreduce-wordcount/input.txt" \
  -output "$OUTPUT" \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 2 \
  -files "mapper.py,reducer.py" \
  -cmdenv "PYTHONIOENCODING=UTF-8"

rm -rf local-output
hdfs dfs -get "$OUTPUT" local-output
cat local-output/part-r-*

Ecosystem and Current Role

Technology How it works Strengths Weaknesses When to use Typical scale
MapReduce Batch map, shuffle, and reduce stages Mature and fault-tolerant High latency; shuffle overhead Large-scale batch processing Throughput scales by adding workers; often shuffle-bound
Spark Memory-centric dataflow engine with shuffle stages Faster iterative and interactive processing Memory pressure and operational complexity Streaming, analytics, and ML Scales by adding workers; network- and memory-sensitive

Hive provides SQL over Hadoop, HBase is a wide-column store, Pig is legacy, Sqoop is a retired Apache project, Oozie manages Hadoop workflows, and ZooKeeper provides coordination. Spark often runs on YARN and reads HDFS. Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses, but distributed storage, locality, scheduling, and fault tolerance remain important concepts.

Common Pitfalls

  • Small files: They consume NameNode metadata and create task overhead; combine files or use compact formats.

  • Unsafe combiners: A combiner must preserve the final result.

  • Skewed keys: One reducer may become a bottleneck; use suitable partitioning or a second sort stage.

  • Excessive concurrency: Too many reducers, containers, or large heaps can overload the cluster.

  • Cloud assumptions: Object storage changes trade-offs but does not remove the need for retries, locality-aware processing, or balanced partitions.

Common follow-ups include: “Why is shuffle a bottleneck?”, “When is a combiner unsafe?”, “How is key skew handled?”, and “How does Hadoop recover from a failed DataNode or task?”

4

How does YARN improve Hadoop resource management?

YARN (Yet Another Resource Negotiator) is Hadoop’s cluster resource-management and application-scheduling layer. It separates resource allocation from individual processing engines, allowing multiple workloads to share a cluster through containers. The main trade-off is improved utilization and workload flexibility in exchange for more operational and configuration complexity.

Read full answer

Applies toApache Hadoop 3.x (YARN)

How YARN Works

YARN manages four principal concepts:

  • ResourceManager: Runs globally, receives application submissions, tracks applications, and uses a pluggable scheduler to allocate resources.

  • NodeManager: Runs on each worker node, starts and monitors containers, and reports available resources and node health.

  • ApplicationMaster: Runs once per application and coordinates its execution. It requests containers from the ResourceManager and asks the NodeManagers to launch tasks; MapReduce, Tez, and Spark each provide their own implementation.

  • Container: A logical allocation of resources, such as memory and vcores, together with the processes running in it. YARN can support additional resource types through pluggable implementations.

A typical application follows this sequence:

  1. A client submits the application to the ResourceManager.

  2. The ResourceManager coordinates allocation of the first container, in which the ApplicationMaster runs.

  3. The ApplicationMaster registers with the ResourceManager and requests additional containers.

  4. The scheduler allocates those containers, preferably on nodes near the required data.

  5. NodeManagers launch and monitor the containers.

  6. The ApplicationMaster coordinates the work and requests more resources when needed.

Why YARN Improves Resource Management

Before YARN, the MapReduce v1 runtime combined cluster resource management with a fixed mapper-slot and reducer-slot model. YARN generalized this into application-level containers managed by configurable schedulers.

Capability MapReduce v1 YARN
Resource model Fixed mapper and reducer slots Flexible allocations of memory, vcores, and extensible resource types
Processing engines Primarily MapReduce MapReduce, Spark, Tez, Flink, and others
Scheduling Job-specific slot allocation Application-level containers and pluggable schedulers
Isolation Limited Per-container resource enforcement; memory and optional cgroup controls
Workloads Mostly batch-oriented Batch, iterative, streaming, and interactive

YARN provides the Capacity and Fair Scheduler policies, hierarchical queues, access-control lists, and resource limits. This lets shared clusters serve multiple teams without allowing one application to consume every node. Vcores represent scheduling shares rather than inherently enforcing strict physical-CPU limits, so deployments requiring CPU quotas should configure appropriate container-executor and cgroup controls.

MapReduce on YARN

A word-count job illustrates the interaction between YARN and MapReduce:

  1. Map: Each mapper reads its input split and emits a key-value pair for every word, such as apple and 1.

  2. Partitioner: The partitioner routes equal keys to the same reducer, normally by hashing the key.

  3. Combiner: An optional combiner performs local partial aggregation before intermediate results cross the network.

  4. Shuffle and sort: The MapReduce framework transfers mapper output to reducers and groups records by key.

  5. Reduce: Each reducer sums the counts for its keys and writes the final result to HDFS.

YARN schedules the map and reduce tasks in containers and tries to place them close to the relevant HDFS blocks. This data locality reduces network traffic, although locality is an optimization rather than a guarantee.

Relationship to Hadoop and Modern Systems

YARN manages computation, while HDFS manages storage. HDFS uses a NameNode for metadata and DataNodes for blocks; common Hadoop 3.x defaults include a 128 MB block size and a replication factor of 3. Hadoop 3.x also supports erasure coding and HDFS high availability with multiple standby NameNodes.

The surrounding ecosystem has distinct roles:

  • MapReduce is Hadoop’s original batch-processing framework and runs on YARN.

  • Hive provides SQL over Hadoop data and commonly uses Tez on YARN for execution.

  • HBase is a separate wide-column database built on HDFS.

  • Spark is an in-memory processing engine that can run on YARN, although standalone and managed-cloud variants also exist.

  • Oozie schedules workflows, while ZooKeeper provides distributed coordination.

  • Pig is now a legacy interface, and Sqoop is a retired Apache project.

Many organizations now use cloud object stores such as S3, GCS, or ADLS with managed Spark and cloud warehouses. Nevertheless, YARN concepts—containers, scheduling, workload isolation, data locality, and fault tolerance—remain important Hadoop interview topics.

Common Pitfalls

  • Confusing components: The ResourceManager is not the HDFS NameNode; YARN manages compute resources, while the NameNode manages filesystem metadata.

  • Assuming vcores provide hard CPU limits: Use appropriate cgroup or container-runtime configuration when strict CPU enforcement is required.

  • Ignoring data locality: Scheduling containers near HDFS blocks reduces network traffic.

  • Using excessive container counts: Tiny containers increase scheduling, monitoring, and JVM overhead.

  • Creating many small files: Numerous small HDFS files increase NameNode metadata and I/O overhead.

  • Misunderstanding retries: YARN detects failures and supports reallocation, while the ApplicationMaster or processing framework normally re-executes failed tasks.

5

What roles do NameNode and DataNode play in HDFS?

HDFS uses a master-worker architecture: the NameNode manages filesystem metadata, while DataNodes store and serve the actual data blocks. Together they provide scalable distributed storage with replication and fault tolerance, but the metadata service requires careful NameNode high-availability configuration.

Read full answer

Applies toApache Hadoop 3.x with HDFS

NameNode and DataNode Responsibilities

Component Primary responsibility Does not do
NameNode Maintains the namespace, directory hierarchy, permissions, block locations, and metadata changes Does not store user file contents
DataNode Stores blocks on local disks, serves reads, participates in write pipelines, and sends heartbeats and block reports Does not independently manage the filesystem namespace

The NameNode persists metadata in fsimage checkpoints and edit logs. DataNodes periodically report their status and hosted blocks, allowing the NameNode to locate data and identify missing or under-replicated blocks.

How Reads and Writes Work

For a read, the client contacts the NameNode for block locations. It then contacts nearby DataNodes directly and reads the blocks while verifying their checksums.

For a write, the client asks the NameNode to create the file and determine target DataNodes using replication and rack-awareness rules. The client connects to the first DataNode, which forms a write pipeline with the other selected DataNodes; packets are forwarded through that pipeline. The NameNode records the resulting locations and confirms block state through subsequent reports.

The NameNode coordinates metadata, but bulk data generally does not pass through it. This keeps data transfer scalable, although the namespace remains a central coordination point.

Replication, Locality, and Fault Tolerance

  • Default block size: dfs.blocksize is 128 MB, reducing metadata relative to much smaller blocks.

  • Default replication: A block normally has three replicas on separate DataNodes with rack awareness.

  • Data locality: MapReduce and other Hadoop workloads are scheduled near the DataNodes holding their input, reducing network traffic.

  • Node failure: After a heartbeat timeout, the NameNode excludes the failed DataNode and schedules missing replicas on healthy nodes.

  • Corruption detection: DataNodes verify stored checksums, and clients also verify checksums while reading.

  • Task retries: MapReduce tasks can be re-executed after failures because replicated input blocks remain available.

Hadoop 3.x also supports erasure coding, which can reduce storage overhead for suitable datasets compared with replication. Its trade-off is additional computation and network activity during recovery and reconstruction.

NameNode High Availability

Hadoop 3.x supports an active NameNode, multiple standby NameNodes, and automatic failover with proper fencing. JournalNodes maintain the shared edit log, which standby NameNodes consume to maintain a synchronized metadata view.

When the active NameNode fails, a standby can take over. High availability still requires tested failover procedures and fencing that prevents two active NameNodes from making conflicting metadata changes.

YARN, MapReduce, and the Hadoop Ecosystem

In a MapReduce word count, input is split into tasks; each mapper emits intermediate word-and-count pairs. An optional combiner aggregates pairs locally, while a partitioner determines the reducer for each key. During the shuffle, framework workers transfer and sort map output by key, and reducers sum the counts for each word. Data locality places mappers near their input blocks.

Component Role
YARN Resource management, with a global ResourceManager, per-node NodeManagers, and an ApplicationMaster for each application
MapReduce Fault-tolerant batch processing based on map, shuffle, and reduce phases
Hive SQL-based analytical processing over data stored in Hadoop
HBase Distributed wide-column NoSQL storage
Pig Legacy high-level data-flow language for Hadoop
Sqoop Retired Apache project formerly used for bulk transfer between Hadoop and relational databases
Oozie Workflow and job scheduling
ZooKeeper Distributed coordination and synchronization
Spark Separate, faster primarily in-memory engine that can run on YARN and read HDFS

Many organizations now use cloud object stores such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage, together with Spark and cloud warehouses. HDFS concepts nevertheless remain important in on-premises Hadoop interviews, especially distributed storage, data locality, replication, and failure recovery.

Common Pitfalls

  • Thinking the NameNode stores file data: It stores metadata; DataNodes store the blocks.

  • Assuming replication is always three: Configuration and erasure coding can change the storage strategy.

  • Ignoring NameNode high availability: A production cluster needs tested failover and fencing.

  • Overlooking the small-files problem: Large numbers of small files consume disproportionate NameNode memory and metadata capacity.

  • Confusing task retries with replication: MapReduce retries failed tasks, while HDFS re-replicates missing or under-replicated blocks.

6

How does rack-aware replica placement work in HDFS, and why does it matter?

Rack awareness is HDFS’s strategy for placing block replicas across different network racks while limiting cross-rack transfers. It matters because the usual replication factor of 3 can survive a single rack failure without placing every replica on a separate rack. The main trade-off is additional cross-rack network use, plus the need for accurate topology and sufficient capacity in multiple racks.

Read full answer

Applies toApache Hadoop 3.x HDFS

How Rack Awareness Works

The HDFS NameNode uses a configured network-topology mapping to determine which DataNodes belong to the same rack. Administrators maintain this mapping through node-to-switch configuration or a topology script; the NameNode does not automatically discover the physical network.

A rack is normally a group of DataNodes connected through the same switch or switch group. A switch, network-component, or power failure can affect multiple nodes in that rack, so replicas confined to one rack provide less failure isolation.

Typical Replica Placement

With the default replication factor of 3 and at least two available racks, HDFS normally places replicas as follows:

  1. First replica: On the client’s local DataNode if the writer runs on a DataNode with sufficient space; otherwise, the NameNode selects another node.

  2. Second replica: On a node in a different rack when one is available.

  3. Third replica: On another node in the same rack as the second replica, when capacity and health permit.

If the preferred nodes are unavailable, HDFS relaxes the placement constraints while respecting configured capacity and rack policies. This avoids using one cross-rack copy for every replica and balances fault isolation against network bandwidth.

Why It Is Important

Aspect Rack-aware placement Rack-oblivious placement
Failure isolation Spreads replicas across racks Can concentrate replicas in one rack
Network usage Limits cross-rack transfers May use more inter-rack bandwidth
Rack-failure resilience A rack failure leaves replicas elsewhere A rack failure could remove all replicas
Write locality Can prefer local and nearby nodes Does not exploit rack topology
Capacity balancing Constrained by available racks Uses simpler node-selection rules

Rack awareness is especially important for large datasets because Hadoop 3.x uses a default HDFS block size of 128 MB and commonly stores each block with a replication factor of 3.

Fault-Tolerance Role

Rack awareness complements HDFS’s other fault-tolerance mechanisms:

  • Block replication maintains multiple copies of each block.

  • Task re-execution lets YARN rerun failed MapReduce or Spark tasks.

  • Rack-aware placement protects against failures affecting an entire rack.

  • Heartbeats identify dead DataNodes, while block reports describe block inventory and missing replicas; the NameNode can then schedule re-replication to satisfy the policy.

Replication protects stored data, while task re-execution restores failed computation. They address different failure modes.

Operational Considerations

Accurate topology information is essential. Stale mappings can place replicas too close together, undermine rack-failure isolation, or increase cross-rack traffic. Placement also becomes less effective when the cluster has only one rack, suitable racks lack capacity, or nodes are decommissioned.

Rack awareness does not solve the small-files problem. Many small blocks still consume NameNode metadata and scheduler resources; compacting them into larger files reduces that overhead. Hadoop 3.x also supports NameNode high availability, with an active NameNode, multiple standby NameNodes, and automatic failover. This protects metadata-service availability but does not replace rack-aware placement of DataNode replicas.

HDFS erasure coding follows the same general topology goal but uses data and parity blocks arranged in stripes, with separate placement policies rather than the replication strategy described above.

Common Pitfalls

  • Confusing replication with rack awareness: Replication creates multiple copies; rack awareness determines where they are placed.

  • Assuming every replica is in a separate rack: The usual third replica shares a rack with the second to reduce cross-rack traffic.

  • Ignoring topology configuration: Incorrect switch mappings undermine isolation and locality.

  • Overlooking capacity constraints: Rack-aware placement cannot be satisfied if suitable racks lack healthy nodes with space.

  • Applying replication assumptions to erasure coding: Erasure-coded data uses data and parity blocks rather than three identical replicas.

7

How does Hadoop differ from traditional RDBMS?

Hadoop is a distributed data-processing framework for storing and analyzing very large datasets across commodity servers. It solves large-scale batch and analytics problems through horizontal scaling, fault tolerance, and flexible data formats, whereas traditional RDBMS systems generally optimize for low-latency queries, structured data, and transactional integrity. Hadoop’s main trade-off is greater scale and throughput at the cost of higher latency and less efficient handling of small, frequently updated datasets.

Read full answer

Applies toHadoop 3.x; traditional RDBMS concepts across all versions

Core Differences

Characteristic Hadoop Traditional RDBMS
Primary use case Large-scale batch processing, analytics, and data lakes OLTP and structured analytical queries
Data model HDFS stores opaque files; ecosystem tools can apply schemas at read time Usually schema-on-write with fixed tables and columns
Scaling Scales horizontally by adding commodity nodes Commonly scales vertically, although distributed RDBMS systems can scale horizontally
Storage HDFS stores files as blocks; replication defaults to three, while erasure coding is optional Managed tablespaces, data pages, indexes, and transaction logs
Processing YARN-managed distributed workloads, commonly MapReduce or Spark A database engine executes queries against local or clustered storage
Latency Usually higher because jobs start processes and scan substantial data Usually lower and predictable for indexed point queries
Transactions HDFS itself has no general multi-row transaction layer Strong ACID transactions are commonly supported
Fault tolerance HDFS replication or erasure coding, plus task re-execution Replication, backups, write-ahead logs, and automatic recovery

Hadoop Architecture

  • HDFS: Stores files as blocks. With Hadoop 3.x defaults, a block is 128 MB and is replicated across three DataNodes unless configured otherwise.

  • NameNode: Maintains metadata such as block locations, permissions, and the directory hierarchy. User data remains on the DataNodes.

  • DataNode: Stores the actual blocks, performs block I/O, and carries out replication and recovery operations under NameNode direction.

  • YARN: Provides resource management through a ResourceManager and NodeManagers. Each application has an ApplicationMaster that coordinates resource requests and task execution.

  • MapReduce: Processes data through map, shuffle-and-sort, and reduce phases. Combiners perform local partial reductions, while partitioners select the reducer for each key.

Hadoop attempts to schedule computation on nodes that already hold the required input blocks. This data locality reduces network transfers and improves throughput.

Data and Ecosystem

HDFS itself has no schema or structured data model, but it commonly supports tools that add one:

  • Hive provides SQL-style access to data in Hadoop and supports schema-on-read workflows.

  • HBase is a distributed, wide-column NoSQL data store.

  • Spark provides faster in-memory and iterative processing and commonly reads HDFS while running on YARN.

  • ZooKeeper coordinates distributed services, while Oozie manages Hadoop workflows.

  • Pig is a legacy data-flow tool, and Sqoop is a retired Apache project.

Raw files can therefore be loaded before their analytical structure is finalized, but good schema and partition design remain necessary. Columnar formats such as Parquet and ORC are usually preferable for analytical queries because they reduce I/O and support predicate pushdown.

Reliability and High Availability

  • Block replication protects against DataNode failures by maintaining copies of each block.

  • Task re-execution allows YARN applications to rerun failed tasks after a node or process failure.

  • Erasure coding, available in Hadoop 3.x, can reduce storage overhead, although recovery may require more network traffic and computation than reading a replica.

  • NameNode high availability in Hadoop 3.x supports one active NameNode and one or more standby NameNodes so metadata service can continue during a primary failure.

Replication generally provides simpler and faster recovery, while erasure coding uses less storage. These mechanisms address different operational trade-offs and can coexist within a cluster.

Advantages, Limitations, and Modern Role

Advantages - Handles datasets and workloads that exceed the practical capacity of a conventional database server. - Scales horizontally on commodity hardware and tolerates component failures. - Accommodates many data formats and supports computation close to stored data.

Limitations - Is not usually suitable for low-latency transactions or frequent small updates. - Suffers from the small-files problem, because each file consumes filesystem metadata and may create scheduling overhead. - Requires operational care involving schemas, partitioning, file formats, resource configuration, and NameNode metadata capacity.

Many organizations now use cloud object stores such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage with Spark or managed cloud warehouses. Nevertheless, HDFS and YARN concepts remain important because they explain distributed storage, data locality, fault tolerance, resource scheduling, and the design trade-offs behind modern lakehouse platforms.

Common Pitfalls

  • Do not describe Hadoop as simply a faster relational database; its strength is distributed scale and throughput.

  • Do not claim that HDFS itself provides ACID transactions or fine-grained updates.

  • Distinguish HDFS storage, YARN resource management, and MapReduce processing.

  • Avoid many tiny files; combine data into appropriately sized files.

  • Do not assume every production cluster uses the default replication factor of three.

Interviewers may ask about NameNode failover, block recovery, combiners, partitioners, data locality, and how Hadoop differs from Spark.

8

How do you secure a Hadoop cluster with Hadoop 3.x mechanisms?

Securing a Hadoop 3.x cluster requires defense in depth: Kerberos establishes identities, authorization limits privileges, encryption protects data, and auditing supports investigation. Together, these controls address impersonation, unauthorized access, disclosure, and accountability. The main trade-off is stronger protection versus more complex ticket, key, policy, and operational management.

Read full answer

Applies toHadoop 3.x (HDFS, YARN, and MapReduce; Hadoop Streaming examples in Python 3.12)

Defense-in-Depth Controls

Mechanism Purpose Important details
Kerberos Authentication Authenticates users and services through time-limited KDC tickets. Protect keytabs, rotate KDC keys securely, and distribute replacement keytabs carefully.
HDFS permissions and ACLs Authorization POSIX permissions, ownership, and HDFS ACLs control file access. Avoid routine use of the HDFS superuser identity.
YARN ACLs and limits Workload control Queue and application ACLs control submission and access, while scheduler limits constrain requested CPU and memory.
Apache Ranger Central policy and auditing Centralizes authorization and auditing for HDFS, Hive, HBase, YARN, and other supported services. Retain audit records in protected monitoring storage.
Apache Knox Gateway security Provides a controlled gateway for supported REST APIs; it complements rather than replaces service-level authorization.
Network controls Reduced exposure Restrict exposed ports, isolate administrative interfaces, and require controlled paths for privileged access.
Encryption in transit Network confidentiality Use HTTPS/TLS for web and REST endpoints. Hadoop RPC and supported data-transfer paths use separately configured SASL protection, so web-interface TLS does not secure RPC by itself.
Encryption at rest Storage confidentiality HDFS encryption zones use KMS-backed keys for file contents and per-file encryption metadata; namespace paths remain unencrypted. Authorized reads decrypt transparently after KMS checks access.

Architecture and Ecosystem Roles

  • HDFS: The NameNode manages metadata and the namespace, while DataNodes hold the blocks. The default block size is 128 MiB, and the default replication factor is 3. Replication and erasure coding provide availability rather than confidentiality.

  • HDFS high availability: Hadoop 3.x supports multiple standby NameNodes and erasure coding. Erasure coding can reduce storage overhead, although reconstruction consumes additional network bandwidth and disk I/O.

  • YARN: The ResourceManager allocates resources, NodeManagers manage worker-node resources, and an ApplicationMaster coordinates each application. Failed tasks can be re-executed, with the scheduler retrying for suitable locality.

  • Ecosystem placement: Hive provides SQL on Hadoop; HBase is a wide-column store; Pig is a legacy data-flow tool; Sqoop is a retired Apache project; Oozie manages workflows; ZooKeeper provides coordination; and Spark is an in-memory engine that often runs on YARN and reads HDFS.

  • Many organizations now use cloud object storage such as S3, GCS, or ADLS and cloud warehouses. Nevertheless, distributed storage, data locality, fault tolerance, workload isolation, and authorization remain important interview topics.

MapReduce and Data Locality

  1. Map: Map tasks process input splits and emit key-value pairs, such as a word and a count of one.

  2. Combiner: An optional map-side combiner aggregates values locally before shuffle traffic. It must preserve the final result; summing counts is safe for word count.

  3. Partitioner: The partitioner assigns each key to a reducer partition, so equal keys reach the same reducer.

  4. Shuffle and sort: Map outputs are transferred and grouped by key for the reducers.

  5. Reduce: Each reducer aggregates the values for its keys and writes the output.

  6. Data locality: YARN schedules tasks near the DataNodes holding their input when containers are available, reducing remote data movement.

Code Example

This example assumes the caller already has a valid Kerberos ticket. The klist command verifies that a ticket is present before Hadoop Streaming runs under the caller’s Hadoop identity.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.split():
        # Hadoop Streaming separates the key and value with a tab.
        print(f"{word}\t1")
# reducer.py
import sys

counts = {}

for line in sys.stdin:
    word, count = line.rsplit("\t", 1)
    counts[word] = counts.get(word, 0) + int(count)

for word in sorted(counts):
    print(f"{word}\t{counts[word]}")
set -euo pipefail

HADOOP_HOME="${HADOOP_HOME:?Set HADOOP_HOME to the Hadoop installation directory}"

# Verify that the caller has a current Kerberos ticket.
klist

# Create deterministic local input and upload it to HDFS.
printf 'a b a\nb c\n' > input.txt
hdfs dfs -mkdir -p "/user/$USER/wordcount/input"
hdfs dfs -put -f input.txt "/user/$USER/wordcount/input/"
hdfs dfs -ls -R "/user/$USER/wordcount/input"

# Locate the installed Hadoop Streaming library.
HADOOP_STREAMING_JAR=$(find \
  "$HADOOP_HOME/share/hadoop/tools/lib" \
  -name 'hadoop-streaming-*.jar' -print -quit)
test -n "$HADOOP_STREAMING_JAR"

# The output directory must not already exist.
OUTPUT="/tmp/wordcount-output-$(date +%s)-$$"
hadoop jar "$HADOOP_STREAMING_JAR" \
  -input "/user/$USER/wordcount/input" \
  -output "$OUTPUT" \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -files mapper.py,reducer.py

# Print the generated word-count pairs.
hdfs dfs -cat "$OUTPUT/part-*"

Common Pitfalls

  • Kerberos is not authorization: Configure HDFS, Ranger, Hive, HBase, and YARN authorization separately.

  • Encryption is not access control: A process with legitimate read and key access can decrypt protected data.

  • Shared service accounts weaken auditing: Give users, jobs, and services distinct identities and protect their keytabs.

  • Replication is not security: Replication and erasure coding primarily provide fault tolerance.

  • HA is not automatically secure: Protect NameNode RPC and web interfaces, ZooKeeper, JournalNodes, and failover tooling with authentication, encryption, and network controls.

  • Small files increase operational risk: They burden the NameNode and scanning processes; combine files safely and monitor file-size distributions.

  • Expect follow-ups about Kerberos ticket lifetime, Ranger policy design, KMS recovery, NameNode high availability, and task re-execution.

9

What role does HBase play in the Hadoop ecosystem?

HBase is a distributed NoSQL wide-column database that stores large tables on HDFS and provides low-latency random reads and writes. It solves the need for online, transactional-style access where HDFS is better suited to sequential file processing. Its main trade-off is strong scalability and write throughput in exchange for limited relational querying and additional operational complexity.

Read full answer

Applies toApache HBase 2.x on Hadoop 3.x

Role in the Hadoop Ecosystem

  • HDFS: Provides durable distributed storage. The NameNode stores metadata, while DataNodes store blocks. The usual Hadoop defaults are a 128 MB block size and a replication factor of 3.

  • HBase: Adds an online database layer over HDFS, organizing data into tables, regions, rows, column families, and column qualifiers.

  • YARN: Manages resources through the ResourceManager, NodeManagers, and ApplicationMasters. HBase normally runs as a persistent service rather than a YARN application.

  • MapReduce: Processes large HDFS datasets with data locality. In word count, mappers emit word-count pairs, partitioners route matching keys together, shuffle and sort group the keys, reducers sum the counts, and optional combiners pre-aggregate data on mapper nodes.

  • ZooKeeper: Provides coordination for HBase, including cluster membership, leader election, and region-assignment metadata.

Architecture and Data Model

  • HBase Master: Coordinates administrative work and assigns regions to RegionServers.

  • RegionServers: Host regions, serve client requests, and host data locally with the corresponding HDFS blocks. Regions can split and move during balancing or recovery.

  • Write path: Updates are recorded in the Write-Ahead Log, maintained in memory in a MemStore, and eventually flushed into persistent, HDFS-backed HFile files.

  • Row key: Determines physical row ordering. A well-designed key distributes requests across regions and reduces the risk of a hot region.

  • Column families: Group related columns and define storage properties such as maximum versions and time-to-live.

  • Sparse storage: Different rows can omit column qualifiers, making the model suitable for optional or evolving attributes.

  • Consistency: HBase provides strong consistency and atomic updates for individual rows. It is not a general multi-row ACID transaction system, and its primary access path is the ordered row key rather than arbitrary secondary indexes.

Ecosystem Placement

Component How it works Strengths Weaknesses Typical scale or use
HDFS Stores immutable files as replicated blocks High throughput and durability Inefficient for random updates Petabyte-oriented batch storage
HBase Stores regions and rows over HDFS Low-latency access and high write throughput Limited joins and SQL; sensitive to row-key design Large online tables partitioned into regions
Hive Translates SQL into execution plans Familiar SQL and strong analytics integration Not designed for low-latency row updates Large analytical datasets and warehouses
Spark Runs in-memory DAG jobs, often on YARN, and reads HDFS Fast iterative, batch, and streaming processing Cluster and memory overhead Cluster-scale ETL and analytics
Pig Uses the legacy Pig Latin data-flow language Compact batch data flows Legacy ecosystem and lower adoption Existing Hadoop batch jobs
Sqoop Uses legacy connectors for bulk RDBMS transfers Batch-oriented import and export Retired Apache project with no active development Legacy migration workflows only
Oozie Schedules Hadoop job DAGs and dependencies Native Hadoop workflow coordination Limited adoption in modern platforms Existing Oozie deployments
ZooKeeper Maintains coordination metadata and watches Reliable distributed coordination Not a general-purpose database Small coordination state for HBase and Hadoop

Current Role and Operational Considerations

  • Cloud alternatives: Many new workloads use S3, Google Cloud Storage, ADLS, Spark, cloud warehouses, or managed NoSQL services. HBase remains relevant for organizations with Hadoop estates that need predictable HDFS-backed random access.

  • Fault tolerance: HDFS replication protects stored blocks, MapReduce re-executes failed tasks, and HBase replays the WAL and reassigns regions after a RegionServer failure.

  • NameNode high availability: Hadoop 3.x supports an active NameNode with one or more standby NameNodes, using a JournalNode quorum and fencing.

  • Small-files problem: Numerous small HDFS files consume NameNode metadata and increase scheduling overhead. Compact files into efficient sequence or columnar formats when appropriate.

Common Pitfalls

  • Poor row-key design: Sequential or low-cardinality keys concentrate requests and undermine scalability.

  • Treating HBase as Hive: Complex joins and unrestricted analytical queries usually fit Hive, Spark, or a warehouse better.

  • Ignoring operational dependencies: Plan for ZooKeeper coordination, WAL recovery, region rebalancing, backups, and NameNode high availability.

  • Confusing storage and access models: HDFS stores files; HBase provides direct row-key access, flexible columns, and versioning.

Likely follow-up questions include: How does row-key design affect hotspotting? What happens during RegionServer recovery? How does WAL replay restore data? When is HBase preferable to Hive, Spark, or a cloud NoSQL service?

10

What is Apache Hive and what problems does it solve?

Apache Hive is a data warehouse and ETL platform for querying large datasets with HiveQL, a SQL-like language. It provides schemas, metadata, query compilation, and distributed execution over data in Hadoop ecosystems. Its main trade-off is scalable batch analytics versus the lower latency and stronger transactions of specialized databases.

Read full answer

Applies toApache Hive 4.1.0 with Hadoop 3.x

Core Architecture

Hive separates SQL access and metadata management from storage and execution:

  • Interfaces: Hive CLI, JDBC, and Thrift clients submit queries.

  • Compiler and optimizer: HiveQL is parsed, validated, optimized, and converted into an execution plan.

  • Metastore: A relational database stores table definitions, partitions, statistics, and other metadata.

  • Execution engines: Hive 4.x runs plans through Apache Tez or MapReduce.

  • Storage: Table data normally resides in files on HDFS.

Problems Hive Solves

Hive primarily addresses large-scale data warehousing and batch analytics:

  • SQL access: Analysts can use SELECT, JOIN, aggregation, and window functions instead of writing MapReduce programs.

  • Schema management: Column types, SerDes, partitions, and file formats provide a consistent view of stored data.

  • ETL processing: HiveQL prepares data for reporting, downstream analytics, and machine-learning workflows.

  • Distributed scale: Queries execute across a cluster rather than requiring all data to fit on one machine.

  • Ecosystem integration: Hive works with HDFS, YARN, Tez, Spark, and related data tools.

  • Metadata discovery: The Metastore provides a shared catalog of datasets and schemas. It does not by itself enforce fine-grained authorization.

Typical workloads include clickstream analysis, log aggregation, business intelligence, fraud analysis, and large-scale data preparation.

Data and Table Layout

  • Columns and types: For example, an event may have event_id BIGINT, event_time TIMESTAMP, and country STRING.

  • File formats: Common choices include ORC, Parquet, and text.

  • SerDes: Serializers and deserializers map Hive types to fields in CSV, JSON, or other file formats.

  • Partitions: Frequently filtered columns become directory names such as country=US/event_date=2025-01-01.

  • Buckets: Data can be divided into a fixed number of buckets to support sampling and certain join optimizations.

Hive manages the data lifecycle of managed tables, so dropping one normally deletes its data. Dropping an external table removes its metadata but leaves the underlying files intact.

Execution on Hadoop

Hive 4.x executes plans through Tez or MapReduce, generally on YARN. The ResourceManager allocates resources, NodeManagers supervise containers, and an ApplicationMaster coordinates each distributed application.

HDFS stores blocks on DataNodes and metadata on a NameNode. Hadoop 3.x supports multiple standby NameNodes for high availability and erasure coding as an alternative to replication. Its standard configuration uses 128 MB blocks with a replication factor of 3. Failed tasks can be rerun, and replicas protect data from DataNode failures.

In MapReduce word count, mappers emit (word, 1) pairs; optional combiners pre-aggregate locally; partitioners route each key; the shuffle and sort stage groups matching keys; and reducers sum the counts. YARN schedules tasks near the nodes holding their inputs to preserve data locality.

Advantages and Limitations

Aspect Strength Limitation
Scale Processes datasets larger than one server Requires partitioning and tuning
Interface Familiar SQL-like HiveQL Not fully compatible with every SQL dialect
Performance Effective for large batch scans High latency for small queries
Transactions Supports Hive ACID tables Unsuitable for high-frequency OLTP workloads

Hive is generally a poor fit for millisecond-response applications, frequent row-level updates, or tightly constrained transactional systems.

Interview Tips

Many teams now use S3, Google Cloud Storage, Azure Data Lake Storage, Spark, or managed cloud warehouses, but HDFS and YARN concepts remain interview staples because they explain distributed storage, data locality, and fault tolerance. In the Hadoop ecosystem, HBase is a wide-column store, Pig is legacy, Sqoop is retired, Oozie handles workflows, ZooKeeper coordinates services, and Spark is a separate in-memory engine that often runs on YARN and reads HDFS.

Know the main distinctions:

  • HDFS stores files; Hive provides SQL semantics and metadata over data.

  • Relational databases provide lower latency and stronger transactional behavior.

  • Spark is an execution engine; Hive is a SQL-oriented data warehouse layer.

Common mistakes include creating excessive small files, over-partitioning, ignoring data skew, choosing legacy MapReduce unnecessarily, treating Hive as an OLTP database, and failing to back up a shared Metastore.

11

Where does Apache Pig fit in the Hadoop ecosystem?

Apache Pig is a high-level data-flow language for ETL and batch processing on Hadoop. It solves the complexity of writing low-level MapReduce code by expressing filters, joins, grouping, and aggregation as data-flow operations. Its main trade-off is concise, high-level processing in exchange for less execution control and limited momentum in new Hadoop projects.

Read full answer

Applies toApache Pig 0.17.0 with Hadoop 3.x

Where Pig Fits

Pig normally sits above Hadoop’s storage, resource-management, and execution layers:

  • HDFS stores input and output data in replicated blocks. The NameNode manages file and block metadata, while DataNodes hold the actual blocks.

  • Pig parses a Pig Latin script into a logical plan describing the desired transformations. Its planner then generates a physical execution plan for the selected execution mode.

  • Traditional execution uses MapReduce, potentially producing multiple jobs for one script. Pig can also run in local mode or use other supported execution modes.

  • YARN provides cluster resources through the ResourceManager and NodeManagers. Each MapReduce application has an ApplicationMaster that coordinates its tasks.

  • Results can be written to HDFS or another configured data store.

MapReduce is easiest to understand through word count:

  1. The mapper emits each word with a count of 1.

  2. An optional combiner performs map-side preaggregation on mapper output, reducing network traffic.

  3. A partitioner chooses the reducer responsible for each key, keeping related keys together.

  4. The framework performs the shuffle and sort, transferring map output to the appropriate reducers and grouping records by key.

  5. The reducer sums the counts for each word.

Hadoop also tries to exploit data locality by scheduling map tasks near the blocks they read. Reducers then retrieve the relevant map output, which can require network transfer.

Pig Latin Capabilities

Pig provides high-level relational and data-flow operators:

  • LOAD and STORE read and write relations.

  • FILTER keeps records that satisfy a condition.

  • FOREACH GENERATE transforms or derives fields.

  • JOIN combines relations using specified keys.

  • GROUP groups records for aggregation.

  • ORDER and LIMIT sort and restrict results.

  • PARTITION divides a relation for parallel processing.

  • User-defined functions provide custom logic when built-in operators are insufficient.

A script describes logical operations; it does not itself contain the generated physical plan.

Hadoop Ecosystem Comparison

Component How it works and strengths Weaknesses Scale and typical use
Apache Pig Expresses ETL flows that traditionally compile to MapReduce Legacy, with limited low-level control Existing batch-ETL and migration workloads
Hive Provides SQL-like warehouse queries over Hadoop data Query-planning and startup overhead SQL analytics at Hadoop scale
HBase Wide-column storage with random row-level access Greater operational complexity Low-latency lookups and updates
Spark Faster in-memory engine, often running on YARN and reading HDFS Memory and cluster-management overhead Iterative, streaming, and mixed analytics
Sqoop Transfers data between relational databases and Hadoop Retired Apache project; maintained connectors require separate work Legacy bulk ingestion
Oozie Coordinates Hadoop workflows and schedules Legacy compared with modern orchestrators Existing Hadoop workflow deployments
ZooKeeper Provides distributed coordination and synchronization Specialized coordination API, not a compute engine Coordination among distributed services

Fault Tolerance and Operations

  • Block replication: Hadoop’s common configuration uses a 128 MB HDFS block size and replication factor of 3, although deployments can change both settings.

  • Task re-execution: YARN can retry failed MapReduce tasks, potentially on another node, until configured retry limits are reached.

  • Erasure coding: Hadoop 3.x can use erasure coding instead of replication for selected HDFS data, reducing storage overhead with different recovery and performance characteristics.

  • NameNode high availability: Hadoop 3.x supports multiple standby NameNodes. The active serves metadata requests while standbys remain synchronized for failover; JournalNodes support shared editing, and automatic failover can use ZooKeeper coordination.

  • Small files: Many tiny files create map-task and metadata overhead. Combining them into appropriately sized sequence files or other efficient formats usually improves performance.

Modern Relevance

Many organizations have moved from Pig toward Spark, cloud warehouses, or managed processing services. Cloud object stores such as S3, GCS, and ADLS also replace HDFS in some architectures, while Spark or cloud compute handles processing. Pig’s primary current role is therefore maintaining existing pipelines, understanding historical Hadoop jobs, or incrementally porting workloads to more actively maintained engines.

Interview Tips

  • Explain that Pig is a data-flow language and execution framework, not HDFS, YARN, or a database.

  • Distinguish the layers: HDFS stores data, YARN manages resources, MapReduce executes jobs, and Pig describes transformations.

  • Remember that traditional Pig execution is batch-oriented and MapReduce-based.

  • Avoid presenting Pig, Sqoop, or Oozie as default choices for new Hadoop projects.

  • Connect Hadoop’s durability and scalability to block replication, task re-execution, data locality, small-file avoidance, and NameNode high availability.

12

How does Apache Flume collect logs and events for Hadoop?

Apache Flume is a distributed agent for collecting logs and events into Hadoop. It routes each event through a source, channel, and sink, decoupling producers from temporarily unavailable destinations. Its main trade-off is simple Hadoop integration versus limited computation and at-least-once delivery that can create duplicates.

Read full answer

Applies toApache Flume 1.12.x, Hadoop 3.x, and Python 3.12

How It Works

Flume runs one or more Java agents. Each event contains a body and optional headers used for routing or filtering.

  • Source: Accepts records from systems such as taildir, netcat, Kafka, or custom source APIs.

  • Channel: Buffers events between source and sink. A memory channel is fast but volatile, a file channel stores events on local disk, and a Kafka channel provides distributed buffering.

  • Sink: Writes events to HDFS, Kafka, Elasticsearch, or another destination.

A typical flow is source → channel → sink. Channel capacity limits buffering; if a channel fills, behavior depends on the source and can include a ChannelException or retry rather than unlimited growth.

Hadoop Integration and Reliability

Flume’s HDFS sink normally writes each running output to a temporary file, then closes and renames that file when it is rolled or closed; it does not wait for the entire ingestion job to finish.

  • NameNode: Stores HDFS metadata and block locations.

  • DataNodes: Store the actual data blocks.

  • Defaults: HDFS normally uses 128 MB blocks and a replication factor of 3.

  • Hadoop 3.x: Supports optional erasure coding and NameNode high availability with multiple standby NameNodes.

  • Fault tolerance: HDFS block replication protects stored data, while Hadoop task re-execution handles failed MapReduce tasks.

Flume’s file or Kafka channels can provide recoverable buffering, but delivery is generally at least once; consumers should tolerate duplicates and use idempotent processing. Run Flume agents near event producers to reduce network traffic, and run Hadoop map tasks near their HDFS input blocks to preserve data locality.

Many organizations now use object storage such as S3, GCS, or ADLS, along with Spark or managed cloud ingestion and warehouse services. Flume nevertheless remains useful where existing Hadoop infrastructure and predictable HDFS delivery are priorities.

Processing and Ecosystem Role

After Flume writes events to HDFS, MapReduce can process them under YARN. A word-count job follows these steps:

  1. Map: Emits each word with a count of 1.

  2. Combiner: Optionally aggregates map output before network transfer.

  3. Partitioner: Selects the reducer for each key.

  4. Shuffle and sort: Groups matching keys by reducer.

  5. Reduce: Emits the final count for each word.

YARN coordinates execution through its ResourceManager, NodeManagers, and per-application ApplicationMaster.

Tool How it works Strength Best for Limitation or scale
Flume Moves events through source-channel-sink agents Simple Hadoop ingestion Logs and events to HDFS Limited computation; scale with agents
Hive Runs HiveQL over Hadoop data Familiar SQL analytics Batch reporting Higher latency than streaming
HBase Uses distributed wide-column tables Random-access row lookups Operational data More complex operations
Pig Uses legacy Hadoop data-flow jobs Supports existing workloads Maintenance only Legacy technology
Sqoop Transfers bulk relational data Straightforward bulk loads Legacy imports Retired project
Oozie Schedules dependent Hadoop jobs Hadoop workflow integration Job DAGs Fewer features than modern orchestrators
ZooKeeper Stores replicated coordination data Leader election and coordination Distributed metadata Not an ingestion system
Spark Runs distributed in-memory transformations and streams Fast large-scale processing Transformations, streaming, and SQL Commonly higher memory demand

Code Example

The following Hadoop Streaming scripts perform word count over events already collected into HDFS.

mapper.py:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

reducer.py:

#!/usr/bin/env python3
import sys

last_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    value = int(value)

    if last_word is not None and word != last_word:
        print(f"{last_word}\t{count}")
        count = 0

    last_word = word
    count += value

if last_word is not None:
    print(f"{last_word}\t{count}")

The -files option distributes the scripts to the task working directories. Configure python3 on the workers to use Python 3.12:

printf 'hadoop logs hadoop\n' > input.txt

hdfs dfs -mkdir -p /user/flume/input
hdfs dfs -put -f input.txt /user/flume/input/
hdfs dfs -ls /user/flume/input

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/mapreduce" -maxdepth 1 \
  -name 'hadoop-streaming-*.jar' | sort | tail -n 1)
OUT="/user/flume/wordcount-$(date +%s)"

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -input /user/flume/input \
  -output "$OUT"

hdfs dfs -cat "$OUT"/part-* | sort

In production, Flume’s HDFS sink would write the collected events to the input location instead of using hdfs dfs -put for test data.

Common Pitfalls

  • Small files: Frequent rolls create many small HDFS files. Tune hdfs.rollInterval, hdfs.rollSize, and hdfs.rollCount, then compact files with a Spark job or controlled Hive rewrite.

  • Memory-channel loss: Memory-channel contents are lost when the JVM fails. Use a file or Kafka channel when recoverability matters.

  • Duplicate events: Design consumers to be idempotent because delivery is at least once.

  • Poor routing: A bad routing key can skew downstream Kafka partitions or MapReduce reducers; Flume routing does not itself select reducers.

  • Data leakage: If logs become machine-learning features, prevent future or test records from entering training data.

  • Confusing transport with computation: Flume primarily moves data; use MapReduce, Spark, or Hive for computation.

13

How does Apache Sqoop integrate with Hadoop?

Apache Sqoop is a command-line tool for bulk transfer between relational databases and Hadoop, primarily HDFS. It solves batch import and export by running JDBC-based MapReduce jobs, with parallel transfers and optional incremental loading. Its main trade-off is efficient bulk movement rather than real-time change-data capture; because Sqoop is a retired Apache project, it is now mainly a legacy Hadoop concept.

Read full answer

Applies toApache Sqoop 1.4.7 (retired) with Hadoop 3.x

Core Capabilities

  • Import: Reads tables from relational databases through JDBC and writes delimited files to HDFS.

  • Export: Reads delimited files from HDFS and writes their records into a relational table.

  • Parallelism: Distributes database reads or writes across multiple MapReduce tasks.

  • Incremental loading: Uses check columns or updated timestamps to import newer records without repeatedly copying the full table.

  • Connector support: Uses database-specific JDBC drivers and connectors, such as those for Oracle, MySQL, PostgreSQL, and SQL Server.

Sqoop is designed for batch workloads. It is not a general-purpose ETL engine, does not perform general row transformations, and is not suitable for low-latency change-data capture.

Hadoop and MapReduce Interaction

HDFS’s NameNode stores namespace metadata, while DataNodes store the actual blocks. In standard Hadoop 3.x replication, the default block size is 128 MB and the default replication factor is 3. Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability, although Sqoop normally interacts directly with HDFS files and directories.

On Hadoop 3.x, Sqoop submits a MapReduce application to YARN. The ResourceManager allocates cluster resources, each NodeManager manages a worker node, and an ApplicationMaster coordinates the application.

A word-count job illustrates the underlying model:

  1. Map: Each input line is converted into (word, 1) pairs.

  2. Combiner: Optional local aggregation reduces repeated words before network transfer. It cannot change the final result.

  3. Partitioner: Selects the reducer for each key, ensuring the same word reaches the same reducer.

  4. Shuffle and sort: Groups and orders intermediate key-value pairs by word.

  5. Reduce: Sums the counts for each word.

  6. Data locality: For HDFS-backed jobs, including the map phase of a Sqoop export, YARN can place tasks near the DataNodes holding their input. A Sqoop import reads from an RDBMS, so its source reads do not benefit from HDFS input locality.

For a Sqoop import, map tasks perform parallel JDBC reads and write results to HDFS. For an export, map tasks read HDFS and reducers perform parallel JDBC writes.

Import and Export Flow

A typical sqoop import connects to a database, selects rows from a table or query, and writes text records into an HDFS target directory. The --hive-import option can also register the imported data with Hive metadata.

A typical sqoop export reads records from HDFS and writes them into a database. Sqoop 1.4.7 supports insert mode, update mode using --update-key, and upsert mode using --merge-key. Because HDFS and the database do not share a transaction, exports should normally target a staging table.

Ecosystem and Modern Role

  • Hive provides SQL access to data stored in HDFS.

  • HBase is a wide-column NoSQL database, often populated through bulk-loading workflows.

  • Pig was a legacy data-flow language for Hadoop.

  • Oozie schedules Hadoop workflows, historically including Sqoop jobs.

  • ZooKeeper coordinates distributed services and leader election.

  • Spark is a faster, in-memory processing engine that commonly runs on YARN and reads HDFS.

Many organizations have replaced HDFS with cloud object storage such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage, and have replaced hand-run Sqoop jobs with managed Spark, ETL services, or cloud warehouses:

Approach How it works Strengths Weaknesses Best fit
Sqoop Parallel JDBC through MapReduce Simple Hadoop bulk transfer Retired; database and JDBC bottlenecks Existing Hadoop estates
Spark Distributed DataFrame and JDBC reads Broad ETL support and structured APIs Cluster tuning and operational complexity Lakehouse and large transformations
Kafka Connect or Debezium Reads database change logs Near-real-time CDC Kafka and connector operations Continuous synchronization
Cloud ETL services Managed connectors and workflows Low platform maintenance Cost and vendor dependence Cloud-native warehouses

Common Pitfalls

  • Retired tooling: Do not recommend a new Sqoop deployment without explaining that maintained alternatives are usually preferable.

  • Small files: Multiple mapper tasks can create many small HDFS files, increasing NameNode metadata and downstream-processing overhead.

  • Duplicate exports: HDFS block replication protects stored data, but a failed task after a database write may be re-executed and duplicate records; use staging tables and idempotent procedures.

  • Inconsistent source data: Concurrent database changes can make one import observe different points in time. Use a consistent snapshot when required.

  • Security: Avoid placing database passwords directly in command history; use protected configuration files or secret management.

  • Interview follow-ups: Expect questions about why Sqoop uses MapReduce, how incremental imports work, where data locality applies, and how its role differs from Spark, Kafka Connect, and Debezium.

14

How does Apache Oozie schedule Hadoop workflows?

Apache Oozie is a server-side workflow scheduler for Hadoop DAGs. It solves dependency ordering, recurring execution, parameterization, and failure handling by coordinating actions such as MapReduce, Hive, Pig, Spark, and shell jobs; its main trade-off is tight Hadoop integration versus a smaller ecosystem and less adoption than modern orchestrators.

Read full answer

Applies toApache Hadoop 3.x with Apache Oozie 5.2.1 (Python 3.12 examples)

How Scheduling Works

Oozie stores workflow and coordinator definitions plus runtime state in a metadata database.

  • Workflow: A DAG of actions connected by dependencies. A start node begins it, an end node completes it, and ok and error transitions determine which nodes run after an action.

  • Action: One executable unit, such as a MapReduce, Hive, Pig, Spark, or shell job.

  • Coordinator: Creates workflow instances according to a calendar, frequency, and execution policy.

  • Bundle: Groups multiple coordinator jobs that should be managed together.

  • Fork and join nodes: Support parallel branches and synchronization points.

  • A user submits a workflow or coordinator definition through the Oozie CLI or web interface.

  • Oozie validates the definition and stores its definition and execution state.

  • For a coordinator job, Oozie starts workflow instances when their schedule becomes eligible.

  • The server determines which actions have satisfied dependencies and launches them through an action executor.

  • For Hadoop jobs, YARN executes them: the ResourceManager allocates containers, NodeManagers host them, and an ApplicationMaster coordinates the application.

  • Oozie monitors completion, follows the appropriate transition, and can retry an action or route it to an error path.

Oozie Expression Language supplies parameters for paths, dates, and job properties without hard-coding environment-specific values.

Hadoop Integration and Fault Tolerance

  • HDFS: The NameNode manages metadata, while DataNodes store blocks. The default block size is 128 MB and the default replication factor is 3. Hadoop 3.x also supports erasure coding as an alternative to replication.

  • MapReduce: A mapper emits each word with a count of 1; a partitioner assigns each key to a reducer partition; the framework shuffles and sorts the map output; and a reducer sums the counts. An optional combiner aggregates counts before data is transferred.

  • Data locality: YARN schedules tasks near the HDFS blocks containing their input when containers are available.

  • Failure recovery: A failed MapReduce task can be re-executed by the Hadoop framework. An Oozie action retry is separate: it resubmits the whole action, so that action must be idempotent or use unique output paths.

  • NameNode HA: Hadoop 3.x supports multiple standby NameNodes, including hot standby configurations. Oozie coordinates workflows above HDFS and YARN rather than replacing their replication or resource management.

Role in Today’s Ecosystem

Tool How it works Strengths Weaknesses Typical scale When to use
Oozie Hadoop-native workflow and coordinator service Direct HDFS and YARN integration Older operational model and smaller ecosystem Small-to-medium DAGs on existing Hadoop clusters Hadoop batch workflows requiring native coordination
Airflow Python-oriented DAG scheduler with its own scheduler and execution services Rich provider ecosystem and flexible task definitions Requires operating Airflow services and its metadata database; dependencies still need careful design Large pipelines with many tasks General data-engineering workflows
Managed cloud orchestrators Provider-hosted scheduling and execution Reduced operational burden and elastic capacity Cost and possible vendor lock-in Elastic cloud workloads Cloud-native batch and data workflows

Hive provides SQL on Hadoop, HBase is a wide-column store, Pig is legacy-oriented, and Sqoop is retired. ZooKeeper provides distributed coordination, while Spark commonly serves as a faster in-memory engine running on YARN and reading HDFS. Many teams now use S3, GCS, ADLS, Spark, and cloud warehouses instead, but HDFS, YARN, and Oozie remain common interview topics.

Code Example

A Hadoop Streaming word-count job can be invoked from an Oozie shell action. This example assumes Python 3.12 is available locally and on the Hadoop NodeManagers.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

The reducer sums counts for adjacent keys in the sorted mapper output.

# reducer.py
import sys
from itertools import groupby


def parse_count(line):
    word, count = line.rstrip("\n").split("\t", 1)
    return word, int(count)


for word, rows in groupby(sys.stdin, key=lambda line: line.split("\t", 1)[0]):
    total = sum(parse_count(line)[1] for line in rows)
    print(f"{word}\t{total}")

Create deterministic input, test locally, and run it on HDFS:

printf 'hadoop runs\nhadoop is reliable\n' > wordcount-input.txt

# Expected output, with a tab between each word and count:
# hadoop<TAB>2
# is<TAB>1
# reliable<TAB>1
# runs<TAB>1
printf 'hadoop runs\nhadoop is reliable\n' \
  | python3.12 mapper.py \
  | sort -k1,1 \
  | python3.12 reducer.py

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/streaming" \
  -name 'hadoop-streaming-*.jar' -print -quit)

hdfs dfs -mkdir -p "/user/$USER/wordcount/input"
hdfs dfs -put wordcount-input.txt "/user/$USER/wordcount/input/"
hdfs dfs -ls -R "/user/$USER/wordcount"

hdfs dfs -rm -r -f "/user/$USER/wordcount/output"
hadoop jar "$STREAMING_JAR" \
  -input "/user/$USER/wordcount/input" \
  -output "/user/$USER/wordcount/output" \
  -files mapper.py,reducer.py \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 1

hdfs dfs -ls "/user/$USER/wordcount/output"
hdfs dfs -cat "/user/$USER/wordcount/output/part-00000"

Common Pitfalls

  • Do not confuse dependency ordering with resource scheduling: Oozie orders actions, while YARN allocates containers.

  • Do not reuse an existing output directory; Hadoop jobs fail when that directory already exists.

  • Make retried actions idempotent or give each attempt a unique output location.

  • Avoid workflows with many tiny actions or files, which increase scheduling overhead and worsen HDFS’s small-files problem.

  • Store credentials through Hadoop credential providers or Oozie’s credential mechanism rather than embedding them in workflow XML.

  • Emphasize Oozie’s DAG model, coordinator scheduling, action dependencies, EL, and separation of responsibilities from YARN and HDFS.

15

What is Apache ZooKeeper's role in Hadoop?

Apache ZooKeeper is a distributed coordination service used to maintain reliable metadata, synchronize services, and support failure detection across multiple machines. In Hadoop, it solves coordination problems such as active-service election and service registration. Its main trade-off is strong consistency and reliability with quorum-based operation, but it adds operational complexity and is not designed to store large application datasets.

Read full answer

Applies toApache ZooKeeper 3.9.x and Apache Hadoop 3.x

Core Capabilities

ZooKeeper organizes its state as a hierarchical namespace of znodes. Clients can create, read, update, and delete znodes while using watches and sessions to react to changes.

  • Persistent znodes: Remain until explicitly deleted.

  • Ephemeral znodes: Disappear when the creating session expires, making them useful for service membership.

  • Sequential znodes: Receive unique increasing suffixes, allowing clients to coordinate ordering.

  • Watches: Notify clients when watched znodes change; a watch is a one-time trigger, not a continuous event stream.

  • Sessions: Detect client failures through heartbeats and expire ephemeral nodes after a timeout.

  • Quorum consistency: A ZooKeeper ensemble continues serving consistent state when a majority of its members are available.

ZooKeeper itself does not implement a generic lock or leader-election service. Applications usually build those abstractions using ephemeral sequential znodes and carefully designed recipes.

Importance to Hadoop

ZooKeeper is especially important for high-availability and coordinated Hadoop ecosystem services.

  • HDFS automatic failover: ZooKeeper helps determine which NameNode should be active and detects the failure of the active NameNode.

  • HBase coordination: HBase uses ZooKeeper for region-server registration, master election, and cluster membership.

  • Oozie workflows: Oozie uses coordination primitives for workflow state, locks, and leader election.

  • Hive and other services: Services can use ZooKeeper for locks, registration, and discovery, although not every deployment requires it.

ZooKeeper does not replace HDFS’s shared-editing and fencing mechanisms. In an HDFS highly available cluster, ZooKeeper-based election must be combined with a shared journal and reliable fencing so that two NameNodes cannot both write to the namespace.

Relationship to Hadoop Components

Component Primary responsibility ZooKeeper’s relationship
HDFS Stores large replicated files; NameNode manages metadata and DataNodes store blocks Supports automatic failover and active NameNode election
YARN Allocates resources through ResourceManager and NodeManager components Usually not required for normal YARN operation
MapReduce Processes data through map, shuffle, and reduce stages Usually does not require ZooKeeper directly
HBase Provides a distributed wide-column database Uses ZooKeeper extensively for coordination
Oozie Schedules Hadoop workflows Uses ZooKeeper for workflow coordination
Spark Provides an in-memory data-processing engine Often uses YARN and HDFS, but normally does not require ZooKeeper

This distinction matters: ZooKeeper is a supporting coordination layer, not Hadoop’s storage engine, resource manager, or processing framework.

Fault Tolerance and Consistency

ZooKeeper uses a replicated ensemble, typically containing an odd number of servers, such as three or five. Writes require agreement from a quorum, preventing independent network partitions from continuing to accept conflicting updates.

Applications also need to account for sessions and retries. A temporary loss of connectivity can expire a client session, remove its ephemeral znodes, and trigger application-level recovery or failover. Clients should therefore use session timeouts appropriately and design restart logic for leader changes.

Modern Relevance and Limitations

Many organizations now use cloud object stores such as Amazon S3, Google Cloud Storage, or Azure Data Lake Storage, together with Spark and cloud warehouses. These systems reduce some traditional Hadoop dependencies, but ZooKeeper remains common in on-premises Hadoop, HBase, Kafka, and other distributed systems.

Its limitations are important:

  • It stores small coordination metadata, not large files or datasets.

  • Every important write depends on quorum availability and network latency.

  • Watch semantics are lightweight but require clients to reread changed state.

  • Distributed locks and election recipes can become difficult to reason about under failures.

  • Session expiration and leader changes must be handled explicitly by applications.

Common Pitfalls

  • Do not describe ZooKeeper as Hadoop’s primary storage layer: HDFS stores data; ZooKeeper coordinates services.

  • Do not claim every Hadoop operation requires ZooKeeper: It primarily supports coordination, service discovery, and selected high-availability features.

  • Do not treat a watch as a reliable event log: It signals that state changed, after which the client must refresh its state.

  • Do not confuse election with fencing: Electing one active service does not prevent a partitioned former leader from continuing to act.

  • Do not store large objects in znodes: This can severely increase ensemble memory, snapshot, and synchronization pressure.

  • Mention quorum availability: A ZooKeeper ensemble tolerates the failure of some members, but it cannot safely accept writes without a majority.

16

How does Apache Spark fit into the Hadoop ecosystem?

Apache Spark is a distributed data-processing engine that often runs on YARN and reads and writes HDFS data. It addresses large-scale batch, streaming, SQL, and machine-learning workloads through DAG-based execution, usually keeping intermediate data in memory where possible. Its main trade-off is lower latency and greater flexibility than MapReduce for suitable workloads, at the cost of greater memory use, tuning, and operational complexity.

Read full answer

Applies toApache Spark 4.x with Hadoop 3.x

Role in the Hadoop Architecture

Hadoop provides complementary layers, while Spark normally occupies the data-processing layer:

  • HDFS: Durable distributed storage. A NameNode manages metadata, while DataNodes store blocks. Hadoop 3.x defaults to 128 MiB blocks and a replication factor of 3, although both are configurable. It also supports erasure coding and high availability with active and standby NameNodes; HDFS Federation can provide multiple standbys across namespaces, while JournalNodes store shared edits.

  • YARN: Resource management. The ResourceManager allocates resources, NodeManagers manage workers, and an ApplicationMaster coordinates each application.

  • MapReduce: Hadoop’s original batch model. It remains supported but is less suitable for iterative or interactive workloads.

  • Spark: A unified execution engine for batch, streaming, SQL, graph, and machine-learning processing. It can use HDFS and YARN, Kubernetes, or standalone clusters.

Spark therefore complements rather than replaces HDFS and YARN: it commonly uses HDFS for durable storage and YARN for scheduling in Hadoop deployments.

How Spark Executes Work

A Spark application has a driver and worker processes called executors. The driver constructs a logical directed acyclic graph (DAG) and submits its execution plan to the cluster manager.

  1. Transformation: Operations define lazy computations over DataFrames, datasets, or RDDs.

  2. Stage construction: Spark creates stages at shuffle boundaries. Narrow dependencies stay within a stage, while joins, aggregations, and other wide dependencies require shuffling data between partitions.

  3. Task execution: Each stage becomes tasks that run in parallel on executors. Spark attempts to place tasks near the data, following Hadoop’s data-locality principle.

Spark SQL and DataFrames use Catalyst for parsing and optimization, enabling techniques such as column pruning and predicate pushdown.

MapReduce and Tool Comparison

A classic MapReduce word count proceeds as follows:

  1. Map: Each mapper tokenizes a line and emits (word, 1) for every token.

  2. Combiner: An optional mapper-side combiner pre-aggregates repeated words to reduce shuffle traffic.

  3. Partitioner: The default partitioner assigns each key to a reducer partition by hash; a custom partitioner can impose another grouping policy.

  4. Shuffle and sort: The framework transfers map output to the assigned reducers and groups values by key.

  5. Reduce: Each reducer sums the counts and writes (word, count) output.

  6. Data locality: The scheduler tries to run map tasks near their HDFS input, reducing network transfers.

Tool How it works Strengths Weaknesses Typical complexity or scale Typical use
Spark Executes DAG-based stages with shuffle operators Batch, streaming, SQL, and ML; low-latency iterative processing Shuffles and memory management can be costly Scales horizontally; cost depends on shuffle volume and partitions General analytics, ETL, streaming, and large-scale ML
MapReduce Runs map and reduce stages through local intermediate output Mature and fault-tolerant High stage I/O; weak fit for iterative algorithms Scales horizontally but incurs costly stage boundaries and I/O Legacy Hadoop batch processing
Hive Provides SQL and metadata over Hadoop data Familiar SQL interface Performance depends on its execution engine Query- and engine-dependent Data warehousing and SQL analytics
Pig Provides high-level Hadoop data-flow scripts Concise batch data flows Legacy and less widely adopted Distributed batch processing with a smaller modern footprint Maintaining existing Hadoop pipelines

Ecosystem Placement and Fault Tolerance

The surrounding components have distinct roles:

  • Hive provides SQL on Hadoop and can use Spark as its execution engine.

  • HBase is a distributed, wide-column NoSQL database for random, low-latency reads and writes.

  • Pig is a legacy data-flow language and platform.

  • Sqoop, formerly used for bulk relational-database transfers, is a retired Apache project.

  • Oozie coordinates scheduled Hadoop workflows.

  • ZooKeeper provides distributed coordination, leader election, and configuration management.

Replication-based HDFS tolerates DataNode failures, while erasure coding uses parity. Spark recovers failed tasks through task re-execution and lineage-based recomputation. Durable results should still be written to HDFS, object storage, or a transactional lake format.

Today’s Role

Many organizations now use Amazon S3, Google Cloud Storage, Azure Data Lake Storage, managed Spark, cloud warehouses, and lakehouse platforms. Spark’s APIs and execution model remain important, while partitioning, shuffle cost, data locality, fault tolerance, and reproducibility remain common interview topics.

Common Pitfalls

  • Ignoring shuffle cost: Repeated joins and aggregations can dominate runtime.

  • Creating excessive small files: Small HDFS files increase metadata pressure and scheduling overhead.

  • Misusing caching: Calling cache() without a clear reuse plan can waste memory.

  • Selecting the wrong cluster size: Executors must have enough memory for their partitions without preventing useful parallelism.

  • Mixing component responsibilities: Spark, HDFS, Hive, and HBase solve different processing, storage, SQL, and database problems.

  • Overlooking data leakage: Fit feature transformations only on training data and use Spark ML Pipeline stages to keep preprocessing within the workflow.

Common follow-up questions include how Spark reduces MapReduce-style intermediate I/O, why shuffles are expensive, how Spark achieves fault tolerance, and when Spark is preferable to traditional MapReduce.

17

How does HDFS recover after a DataNode failure?

HDFS fault tolerance detects an unresponsive DataNode through heartbeats and reconstructs missing block replicas on healthy nodes, allowing stored data to remain available after an individual node fails. With the default replication factor of 3, HDFS trades additional storage and network traffic for fault tolerance; Hadoop 3.x erasure coding reduces storage overhead but introduces more complex, compute-intensive recovery.

Read full answer

Applies toHadoop 3.x (HDFS and YARN)

How the Failure Is Detected

  • DataNode heartbeats: Each DataNode periodically sends a heartbeat to the NameNode.

  • Block reports: DataNodes also report the blocks they store. These reports maintain the NameNode’s view of block locations and replication status.

  • Failure detection: If the NameNode stops receiving a DataNode’s heartbeats for the configured timeout, it excludes that node from healthy-node lists.

  • Impact on blocks: The NameNode uses its block-location metadata to identify blocks that now have too few replicas and marks them as under-replicated.

  • NameNode HA: A DataNode failure does not normally require a NameNode failover. NameNode failover is a separate recovery process for an unavailable active NameNode.

How Blocks Are Restored

For a replicated block, recovery proceeds as follows:

  1. The NameNode identifies the block replicas that were stored on the failed DataNode.

  2. HDFS selects a healthy DataNode with enough disk capacity and a placement permitted by the cluster’s rack-awareness policy.

  3. The selected node receives another copy from an existing replica, using a network-efficient source where possible.

  4. The block remains under-replicated until its configured replication factor is restored.

The default Hadoop 3.x configuration uses 128 MB blocks and a replication factor of 3. If available capacity or placement constraints prevent restoration, HDFS continues trying and the blocks remain under-replicated. If every available copy of a block is lost, HDFS cannot reconstruct it through replication alone.

Erasure Coding and NameNode High Availability

In Hadoop 3.x, HDFS can protect files with erasure coding instead of full replication. Data and parity shards are distributed across DataNodes according to the configured policy. When a shard is missing, HDFS reconstructs it from the surviving data and parity shards, provided enough shards remain. This lowers storage overhead but requires additional network activity and computation during recovery. Administrators can also convert erasure-coded files to replicated storage when that tradeoff is preferable.

Hadoop 3.x high-availability deployments can also use multiple standby NameNodes. If the active NameNode fails, the HA controller must fence it before promoting a standby. This is independent of recovering blocks after a DataNode failure.

MapReduce Recovery

A failed DataNode may be colocated with a failed YARN NodeManager. When a container or task attempt fails, YARN notifies the ApplicationMaster, which requests replacement resources from the ResourceManager. The failed map task can then run again; lost map output is regenerated before being sent to the reducer.

A MapReduce word-count job follows these stages:

  1. The mapper emits each word with a count of 1.

  2. The optional combiner performs map-side aggregation.

  3. The partitioner normally assigns each word to a reducer by hashing its key.

  4. The shuffle and sort phase groups records by key.

  5. The reducer sums the counts for each word.

Common Pitfalls

  • Assuming replication is a complete backup: Simultaneous failures or accidental deletion can still affect multiple replicas, so rack awareness and capacity planning remain important.

  • Ignoring under-replicated blocks: Persistent warnings can indicate insufficient capacity, unavailable target nodes, or restrictive placement policies.

  • Confusing storage and computation recovery: HDFS restores missing blocks, while the ApplicationMaster coordinates failed task attempts and replacement resources.

  • Overlooking the small-files problem: Many small files increase NameNode metadata overhead and reduce parallel-processing efficiency.

  • Ignoring current alternatives: Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses, but HDFS replication, data locality, fault tolerance, NameNode HA, and YARN task recovery remain important interview topics.

18

How does HDFS replicate data blocks?

HDFS replication stores each data block on multiple DataNodes so that hardware failures do not cause data loss. The NameNode selects the replica locations, while the client sends data through a replication pipeline; the main trade-off is greater durability and availability at the cost of additional storage and network traffic.

Read full answer

Applies toApache Hadoop 3.x (HDFS and MapReduce)

Hadoop Architecture Context

  • HDFS provides distributed storage: the NameNode manages metadata and block locations, while DataNodes store the actual blocks.

  • YARN manages resources through its ResourceManager, NodeManagers, and ApplicationMaster. MapReduce is the original batch-processing model.

  • Hive provides SQL, HBase provides a wide-column store, Oozie provides workflows, and ZooKeeper provides coordination. Pig is legacy, while Sqoop is a retired Apache project.

  • Spark is a faster, commonly in-memory processing engine that often runs on YARN and reads from HDFS.

Replication Pipeline

  1. For a new file, the client sends a create request containing the file path, replication factor, and block size to the NameNode. For an append, the client instead obtains the locations of the existing final block and writes through that block’s replica pipeline.

  2. The NameNode selects target DataNodes based on rack topology, available capacity, node availability, and current replication state.

  3. The NameNode returns the ordered replica locations. With the default replication factor of three, the typical rack-aware placement uses a local replica when possible, a second replica in a different rack, and a third replica in the same rack as the second.

  4. The client connects the DataNodes into a write pipeline, normally: Client → DataNode 1 → DataNode 2 → DataNode 3.

  5. The client sends the block only to the first DataNode. Each DataNode forwards the data to the next replica while acknowledgements travel back through the pipeline.

  6. Packets contain checksums so receivers can detect corruption, and failed acknowledgements cause retransmission. Once the required replicas have acknowledged the data, the block is safely replicated; DataNode reports let the NameNode update its replica metadata.

  7. DataNodes periodically send block reports and heartbeats so the NameNode can track replica status.

Hadoop 3.x defaults to a 128 MiB block size and a replication factor of 3, although production clusters can override both settings.

Fault Tolerance and Recovery

  • If a DataNode stops responding, the NameNode marks it dead and identifies under-replicated blocks.

  • A healthy replica is copied to a replacement DataNode, a process called re-replication.

  • Corrupt blocks are detected through checksums and replaced from a healthy replica.

  • HDFS supports multiple standby NameNodes for metadata high availability.

  • MapReduce tasks that fail because data is unavailable can be re-executed on another node.

  • Replication is not the same as a backup: all replicas may still be affected by cluster-wide failures, accidental deletion, or software defects.

Replication Versus Erasure Coding

Hadoop 3.x also supports erasure coding, particularly for cold data. Instead of keeping complete block copies, it stores data and parity blocks in stripes across multiple DataNodes and reconstructs missing data during reads.

Approach Strengths Weaknesses Typical use
Block replication Simple, low read overhead, strong locality Higher storage and network cost Frequently accessed data
Erasure coding Lower storage overhead More complex recovery, higher CPU and network cost during recovery Cold or archived data

Many teams now use cloud object stores such as S3, GCS, or ADLS, but HDFS concepts remain important because they illustrate distributed storage, data locality, replication, and fault tolerance.

Common Pitfalls

  • Confusing replication with backup: replicas share the same failure domain and do not protect against every mistake.

  • Ignoring rack awareness: poor placement can reduce failure tolerance or network performance.

  • Creating many small files: each file still consumes NameNode metadata and can increase processing overhead.

  • Assuming erasure coding behaves like replication: recovery has different CPU, network, and read-path costs.

  • Forgetting to split training data: when processing Hadoop-hosted ML data, fit preprocessing only on training folds to prevent leakage.

Interviewers commonly ask how replica placement balances locality and rack-level failure tolerance, how re-replication works, and why erasure coding is not always suitable for hot data.

19

What is speculative execution in Hadoop MapReduce?

Speculative execution in Hadoop MapReduce launches a duplicate of a slow map or reduce task while the original attempt continues. It reduces job completion time when heterogeneous hardware, overloaded nodes, or skewed data create a straggler. The main trade-off is additional resource consumption: speculation can shorten a job but may waste CPU, memory, disk I/O, and network bandwidth.

Read full answer

Applies toApache Hadoop 3.x MapReduce on YARN

How It Works

  • The MapReduce scheduler compares a running task’s progress with other tasks of the same type and identifies one that appears abnormally slow.

  • YARN may allocate another container for a speculative task attempt of the same logical task.

  • Both attempts process the same input partition and must produce equivalent results.

  • When one attempt completes successfully, MapReduce’s output-commit protocol determines which attempt’s output is retained. Redundant attempts are cancelled when possible, and their temporary output is ignored or cleaned up.

  • Speculation does not replace normal failure recovery. Task re-execution handles failures, while speculation addresses attempts that are healthy but slow.

Role in Hadoop Architecture

  • HDFS stores input and output data as replicated blocks. The NameNode stores metadata, while DataNodes hold the blocks. The default block size is 128 MiB, commonly documented as 128 MB, and the default replication factor is 3. Hadoop 3.x also supports erasure coding and active-standby NameNode high availability with multiple standby NameNodes.

  • YARN provides cluster resources. The ResourceManager allocates containers, NodeManagers run them, and the MapReduce ApplicationMaster coordinates task scheduling and execution.

  • MapReduce divides work into map and reduce tasks. Speculation operates at this task level; it is separate from HDFS block replication.

  • HDFS data locality normally schedules tasks near their input. A speculative attempt may run elsewhere when suitable capacity is available.

  • Ecosystem context: Hive provides SQL over Hadoop data, HBase is a wide-column store, Pig is a legacy data-flow tool, Sqoop is a retired Apache project, Oozie manages workflows, and ZooKeeper provides coordination. Spark is a separate, faster in-memory processing engine that often runs on YARN and reads HDFS.

  • Current role: Many teams now use cloud object storage, managed Spark, or cloud warehouses instead of operating Hadoop directly. Nevertheless, HDFS and YARN concepts such as distributed storage, data locality, and fault tolerance remain common interview topics, while speculative execution remains specific to Hadoop MapReduce task scheduling.

Mechanism Primary purpose
HDFS block replication Preserve data after disk or node failures
Task re-execution Recover from a failed task attempt
Speculative execution Reduce latency caused by slow task attempts
YARN preemption Reclaim resources for higher-priority work

Why Hadoop Uses It

A job finishes only after its slowest remaining task finishes. Without speculation, one slow reduce task receiving unusually frequent keys can hold up an entire job even though the cluster has idle capacity.

Common causes of stragglers include:

  • Slower disks or temporarily overloaded nodes

  • Virtual-machine performance variability

  • Network or filesystem congestion

  • Uneven data distribution and reducer skew

  • Large map partitions that take longer to process

Speculation is most useful when the cluster has spare resources and task duration varies significantly. It is less valuable when the whole cluster is overloaded or when the extra computation delays otherwise healthy tasks.

MapReduce Word-Count Flow

For a word-count job:

  1. The mapper emits each word and the value 1.

  2. An optional combiner performs local aggregation to reduce shuffled data.

  3. The partitioner assigns each key to a reducer, normally by hashing the key.

  4. The shuffle and sort stage groups values by key and moves them to the appropriate reducer.

  5. The reducer sums the values for each word.

Data locality reduces network traffic by scheduling tasks near HDFS input. Speculation can improve reducer-tail latency when one partition contains disproportionately many keys.

Common Pitfalls

  • Confusing speculation with failover: Speculation reduces tail latency; task re-execution recovers from failures.

  • Ignoring data skew: Correcting an excessively large key group may be more effective than enabling speculation.

  • Overloading the cluster: Duplicate attempts consume resources and can slow healthy tasks.

  • Creating side effects: Map and reduce tasks should be deterministic and avoid external side effects because duplicate attempts may execute.

  • Neglecting the small-files problem: Many tiny HDFS files increase NameNode metadata and listing overhead as well as scheduling overhead; compacting or combining files is usually preferable to speculative copies.

  • Assuming every duplicate finishes: The scheduler must have spare capacity and suitable containers.

  • Common interview follow-ups: How do you distinguish speculation from preemption, when would you disable it, and how would you detect reducer skew using task metrics and counters?

20

How does Hadoop process large datasets?

Large datasets in Hadoop are processed by storing blocks in HDFS, allocating compute with YARN, and running parallel tasks with MapReduce or engines such as Spark. This design favors horizontal scalability and fault tolerance, at the cost of operational complexity and batch-processing latency.

Read full answer

Applies toApache Hadoop 3.x (Python 3.12 for the Streaming example)

Core Architecture

  • HDFS: Distributed storage for very large datasets.

  • The NameNode manages the namespace, permissions, and block metadata.

  • The DataNodes hold the actual data blocks and exchange heartbeats and block reports with the NameNode.

  • The default block size is 128 MiB, and the default replication factor is 3.

  • Clients ask the NameNode for block locations and transfer data directly to or from DataNodes.

  • YARN: Hadoop’s resource-management layer.

  • The ResourceManager allocates resources among applications.

  • The NodeManagers monitor workers and launch and monitor containers.

  • Each application has an ApplicationMaster, which coordinates its tasks and requests resources.

  • MapReduce: The original distributed batch model, which processes key-value records through map and reduce tasks.

MapReduce Processing

A word-count job follows these steps:

  1. Split: InputFormat creates input splits, generally aligned with HDFS block boundaries.

  2. Map: Each map task reads its split and emits records such as ("hadoop", 1).

  3. Combine: An optional combiner performs map-side pre-aggregation, such as partially counting repeated words.

  4. Partition: The default hash partitioner routes each key to a reducer.

  5. Shuffle and sort: The framework transfers map output to reducers and sorts it by key.

  6. Reduce: Each reducer combines the values for its keys and writes the results.

Map tasks are normally scheduled near the HDFS blocks containing their input, a behavior called data locality.

Hadoop Ecosystem

Component Purpose Current role
Hive SQL-based analysis over Hadoop data Common for large-scale analytical queries
HBase Low-latency random access through a wide-column store Useful for sparse data and online services
Pig Legacy data-flow language for batch pipelines Often replaced by Spark, Hive, or modern tools
Sqoop Imports and exports relational data Retired from the Apache project
Oozie Hadoop workflow scheduling A legacy alternative to newer orchestrators
ZooKeeper Distributed coordination Supports leader election, configuration, and coordination
Spark Faster, primarily in-memory processing Often runs on YARN and reads HDFS

Many teams now use cloud object storage such as S3, Google Cloud Storage, or Azure Data Lake Storage with managed Spark or cloud warehouses. HDFS and YARN concepts nevertheless remain important because they illustrate distributed storage, data locality, scheduling, and fault tolerance.

Fault Tolerance and Scalability

  • Block replication: HDFS continues serving data when a DataNode fails because other replicas remain available.

  • Task retries: NodeManagers report failed attempts, and the ApplicationMaster coordinates retries. YARN may also launch speculative attempts for slow tasks.

  • NameNode high availability: Hadoop 3.x supports one active and one or more standby NameNodes. JournalNodes maintain shared edits so a standby can be promoted after a failover.

  • Erasure coding: Hadoop 3.x can distribute encoded data and parity across DataNodes, reducing storage overhead compared with three-way replication at the cost of more recovery work.

  • Small files: Many tiny files increase NameNode metadata and scheduling overhead. Compact records into fewer larger Parquet or Avro files, or package them with sequence files or HAR files.

  • Skew and scaling: Uneven key distribution can overload particular reducers, so adding nodes does not guarantee proportional performance gains.

Code Example

This Hadoop Streaming program implements a mapper, a map-side combiner, and a reducer. The combiner aggregates partial counts and emits them in key order.

import sys
from collections import Counter
from itertools import groupby


def main() -> None:
    mode = sys.argv[1]

    if mode == "map":
        for line in sys.stdin:
            for word in line.split():
                print(f"{word}\t1")

    elif mode == "combine":
        counts = Counter()
        for line in sys.stdin:
            word, value = line.rstrip("\n").split("\t", 1)
            counts[word] += int(value)

        for word, total in sorted(counts.items()):
            print(f"{word}\t{total}")

    elif mode == "reduce":
        records = (
            line.rstrip("\n").split("\t", 1)
            for line in sys.stdin
        )
        # The framework sorts reducer input by key.
        for word, group in groupby(records, key=lambda row: row[0]):
            total = sum(int(value) for _, value in group)
            print(f"{word}\t{total}")

    else:
        raise ValueError(f"Unknown mode: {mode}")


if __name__ == "__main__":
    main()

The following commands create and upload the input, distribute wordcount.py with -files, and run the job:

printf 'hadoop hadoop big data\nbig data pipelines\n' > input.txt

hdfs dfs -mkdir -p /wordcount
hdfs dfs -put wordcount.py input.txt /wordcount/
hdfs dfs -ls /wordcount

hadoop jar "$HADOOP_HOME"/share/hadoop/tools/lib/hadoop-streaming-*.jar \
  -input /wordcount/input.txt \
  -output /wordcount/output \
  -files wordcount.py \
  -mapper "python3 wordcount.py map" \
  -combiner "python3 wordcount.py combine" \
  -reducer "python3 wordcount.py reduce"

hdfs dfs -cat /wordcount/output/part-*

Common Pitfalls

  • Ignoring data locality: Scheduling computation far from the relevant blocks increases network traffic.

  • Poor file sizing: Tiny files overload metadata management, while one extremely large file can limit balanced parallelism.

  • Data skew: A few hot keys can make selected reducers much slower than the others.

  • Non-idempotent side effects: Retried tasks can duplicate external writes unless the workflow uses checkpoints, transactions, or idempotent operations.

  • Wrong workload assumptions: HDFS and MapReduce favor large analytical batches, not transaction-heavy, low-latency serving.

  • Missing runtime dependencies: Streaming containers need a compatible python3 executable installed independently of the cluster’s driver nodes.

21

Why are input splits important in MapReduce jobs?

An input split is the logical unit of input processed by a map task, so it determines how MapReduce divides work for parallel processing. Well-chosen splits balance parallelism and scheduling overhead while preserving data locality. The main trade-off is that smaller splits increase task count, whereas larger splits can leave resources idle or create stragglers.

Read full answer

Applies toHadoop 3.x MapReduce

What an Input Split Does

  • Logical division: InputFormat divides the input into InputSplit objects. File-based formats such as FileInputFormat commonly produce FileSplit objects containing a file path, starting offset, and length.

  • Map-task unit: Hadoop normally creates one map task for each split. The number of input splits therefore strongly influences map parallelism.

  • Record processing: A RecordReader presents records within a split to the mapper as key-value pairs.

  • Shuffle input: Each map task sends its intermediate results to reducers through the shuffle-and-sort stage.

An input split is not necessarily identical to an HDFS block. They are often aligned, but InputFormat ultimately determines split boundaries.

HDFS, YARN, and Data Locality

In HDFS, the NameNode stores filesystem metadata, while DataNodes store blocks. Hadoop 3.x uses a default HDFS block size of 128 MB and a default replication factor of 3. Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability.

YARN contains a ResourceManager, NodeManagers, and a per-application ApplicationMaster. The scheduler attempts to run each map task on a node that already holds a replica of the relevant input data. This is data locality: computation moves to the data instead of moving large blocks across the network.

If a task fails, YARN can re-execute it, usually using another replica of the same HDFS block. Consequently, input splits should be large enough to amortize task startup but not so large that failures require substantial recomputation.

MapReduce Word-Count Flow

For word count, the stages are:

  1. Map: Each map task reads one split and emits (word, 1) pairs.

  2. Combiner: An optional local reducer aggregates repeated words within a map task, reducing shuffle traffic.

  3. Partitioner: A hash-based partitioner assigns each word to a reducer.

  4. Shuffle and sort: Reducer inputs are transferred and grouped by key.

  5. Reduce: Each reducer sums the counts for the words assigned to it.

The split size affects the first stage most directly. Too many splits create scheduling, process-launch, and resource-management overhead; too few reduce parallelism and can produce long-running stragglers.

Tuning, Pitfalls, and Modern Relevance

  • Small files: Many tiny files can create many splits and map tasks. Combine files or use suitable input formats when possible.

  • Uneven splits: Skewed data or poorly chosen boundaries can leave some map tasks much slower than others.

  • Memory pressure: Large splits may consume more mapper memory. mapreduce.input.fileinputformat.split.maxsize and related settings can control file splitting.

  • Data leakage: This concern applies mainly to ML pipelines, but feature preprocessing should still process training and validation data separately.

  • Cloud shift: Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses instead of HDFS. HDFS, YARN, split sizing, data locality, and fault tolerance nevertheless remain important Hadoop interview topics.

  • Ecosystem roles: Hive provides SQL over Hadoop data, HBase is a wide-column store, Pig is legacy, Sqoop is retired, Oozie manages workflows, and ZooKeeper provides coordination. Spark is commonly the faster in-memory alternative.

Interview Tips

Be ready to explain that one split normally becomes one map task, that input splits differ from HDFS blocks, and that split sizing controls parallelism and overhead. Common follow-up questions concern combiners, partitioners, shuffle volume, small files, data locality, task retries, and NameNode high availability.

22

What is the role of partitioning in Hadoop MapReduce?

Partitioning in Hadoop MapReduce determines which reducer partition receives each key-value pair produced by a mapper. It enables key-based aggregation while distributing work across reducers, with the main trade-off being reducer parallelism versus shuffle overhead and data-skew risk.

Read full answer

Applies toHadoop 3.x MapReduce and Hadoop Streaming

How It Works

  1. Each mapper emits key-value pairs. In word count, a mapper might emit hadoop\t1.

  2. A partitioner selects the destination reducer partition for each map output.

  3. The default HashPartitioner uses the key’s hash code modulo the number of reducers. Equal keys therefore go to the same reducer partition, although different keys can hash to the same partition.

  4. During the shuffle, the framework transfers map output to the appropriate reducers. Map output is sorted by partition and key, allowing each reducer to process related keys together.

  5. Each reducer reads its partition, groups records by key, and emits the final result. In word count, it sums the counts for each word.

The number of reducers is configured when the job starts. Too many reducers create coordination overhead and many small output files; too few limit parallelism and can worsen skew. A map-only job bypasses reducer partitioning and writes its map output directly.

When Partitioning Is Used

Partitioning is used whenever a MapReduce job has a reduce phase, including:

  • Grouping and aggregation, such as word count, totals, and averages.

  • Grouping records by a business key.

  • Reduce-side joins and co-grouping by a join key.

  • Sending related map output to the same reducer.

  • Custom routing for heavily skewed keys.

A custom org.apache.hadoop.mapreduce.Partitioner<KEY, VALUE> can override getPartition to implement application-specific routing. It must route identical keys consistently when the reducer depends on them being grouped together.

Partitioning Strategies and Combiners

Mechanism How It Works Strengths Weaknesses Typical Use
HashPartitioner Hashes each key into a reducer partition Simple and groups equal keys Hot keys can overload one reducer General aggregation
Custom partitioner Applies application-defined routing Handles specialized grouping or skew Requires correct, deterministic logic Specialized joins and routing
Salting Adds a salt to distribute a hot key across several partitions Mitigates extreme key skew Requires unpacking or a second aggregation stage Extreme key skew
Combiner Performs optional map-side aggregation before shuffle Reduces network transfer Sees only part of the input and cannot replace a reducer Word count and local sums

A combiner is not a partitioner. It is an optional map-side optimization that reduces shuffle volume, but it cannot replace a reducer because each mapper has only a subset of the input.

HDFS, YARN, and Data Locality

  • HDFS stores input as blocks. The NameNode manages metadata, while DataNodes store the default 128 MB blocks with a default replication factor of 3.

  • YARN’s ResourceManager allocates resources, NodeManagers run tasks, and the ApplicationMaster coordinates the MapReduce job.

  • Map tasks are scheduled near the HDFS blocks containing their input, following data locality.

  • Reducer tasks commonly require output from many map tasks, so substantial data still moves across the network.

  • A reducer partition is a logical MapReduce partition, not an HDFS block. Replication, task re-execution, erasure coding, and NameNode high availability do not change reducer partitioning.

  • Hadoop 3.x supports erasure coding and multiple standby NameNodes for high availability.

Many teams now use Spark, cloud object stores such as S3, GCS, and ADLS, or cloud warehouses instead of Hadoop MapReduce. These systems do not use Hadoop’s exact Partitioner contract, although distributed shuffle, data locality, and skew remain related concerns.

Common Pitfalls

  • Ignoring skew: A common key can send nearly all records to one reducer. Measure the distribution and use salting or a custom partitioner when necessary.

  • Breaking key grouping: Sending identical keys to different reducers prevents correct aggregation.

  • Confusing partitioning with locality: Partitioning controls reducer routing, not where input blocks are stored.

  • Misusing combiners: A combiner must preserve the final result even when records are reordered or only partially observed.

  • Creating too many reducers: This increases scheduler, network, and small-file overhead.

  • Using nondeterministic partitioners: Task retries should assign the same data to the same reducer.

Interviewers often ask how skewed keys are handled, why reducers require shuffle traffic, how reducer partitions differ from HDFS blocks, and how the partitioner affects output-file count.

23

How do reducers work in MapReduce and interact with the shuffle phase?

Reducers are MapReduce tasks that consume grouped key-value records after the shuffle and produce final output. They enable distributed aggregation without collecting all results in one process, with the main trade-off being parallel throughput versus data skew, memory pressure, and coordination overhead. A reducer never calls mappers directly: the framework’s shuffle and sort stages route, transfer, and organize mapper output before delivering it.

Read full answer

Applies toApache Hadoop 3.x, Hadoop Streaming with Python 3.12

MapReduce Data Flow

A MapReduce job processes data through these stages:

  1. Input splitting: YARN assigns an ApplicationMaster to coordinate the job.

  2. Mapping: Map tasks process input splits and emit intermediate key-value pairs.

  3. Partitioning: A partitioner assigns each key to a reducer partition.

  4. Shuffle and sort: Map outputs are copied to the appropriate reducer and merged in sorted order.

  5. Reducing: Each reducer receives one partition and processes records grouped by key.

  6. Output: Reducers write final output to HDFS.

Component Responsibility Effect on output
Mapper Converts raw input into key-value pairs Produces intermediate records
Combiner Performs optional local aggregation Reduces data transferred during shuffle
Partitioner Chooses the reducer partition for a key Controls distribution and parallelism
Reducer Aggregates values for each key Produces the final result

Reducer and Shuffle Interaction

The shuffle has two important sides:

  • Map-side shuffle: Mapper output is buffered, partitioned, sorted, spilled to disk, and optionally processed by a combiner. Multiple spill files are merged before transfer.

  • Reduce-side shuffle: YARN starts reducers as resources become available. The framework transfers the relevant map-output partitions and merges their sorted files.

Because the reducer input is sorted by key, all values associated with one key appear consecutively. The reducer tracks the current key, accumulates its values, and emits the result when the key changes or input ends.

For word count, a mapper might emit ("hadoop", 1) repeatedly. The default partitioner sends every "hadoop" record to the same reducer, allowing it to emit ("hadoop", 7).

Reducers and combiners should generally be associative and commutative because the framework may execute them in different orders or call combiners multiple times. A poorly designed partitioner can send most keys to one reducer, creating a hot-key bottleneck.

Code Example

The following Hadoop Streaming mapper and reducer read standard input and emit tab-separated key-value pairs.

# mapper.py
#!/usr/bin/env python3

import sys

for line in sys.stdin:
    for word in line.split():
        # Emit one intermediate occurrence for each word.
        print(f"{word}\t1")


# reducer.py
#!/usr/bin/env python3

import sys

current_word = None
total = 0

for line in sys.stdin:
    word, count_text = line.rstrip("\n").split("\t", 1)
    count = int(count_text)

    # Emit the previous group when the sorted input changes keys.
    if current_word is not None and word != current_word:
        print(f"{current_word}\t{total}")
        total = 0

    current_word = word
    total += count

if current_word is not None:
    print(f"{current_word}\t{total}")

Run the example with Hadoop Streaming:

cat > input.txt <<'EOF'
the quick brown fox
the lazy dog jumps
the fox sleeps
EOF

HDFS_DIR="/user/$USER/wordcount"

hdfs dfs -mkdir -p "$HDFS_DIR/input"
hdfs dfs -put input.txt "$HDFS_DIR/input/"
hdfs dfs -ls "$HDFS_DIR/input"

# Select the Hadoop Streaming JAR installed on this cluster.
STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' -print -quit)"

hadoop jar "$STREAMING_JAR" \
  -D mapreduce.job.reduces=2 \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -input "$HDFS_DIR/input" \
  -output "$HDFS_DIR/output"

hdfs dfs -ls "$HDFS_DIR/output"
hdfs dfs -cat "$HDFS_DIR/output/part-*"

# Deterministic final counts:
# brown=1, dog=1, fox=2, jumps=1, lazy=1, quick=1, sleeps=1, the=3
# The order of output lines depends on reducer partitioning.

Python 3.12 must be installed and accessible on every cluster node that executes the streaming tasks. The output directory must not already exist because Hadoop creates it for the job.

Fault Tolerance and Scale

  • HDFS: The NameNode stores metadata, while DataNodes store blocks. The default HDFS block size is 128 MB and the default replication factor is 3.

  • YARN: The ResourceManager allocates cluster resources, NodeManagers manage individual nodes, and an ApplicationMaster coordinates each application.

  • Task recovery: Failed map or reduce task attempts can be re-executed on another node, subject to the job’s retry limits.

  • Hadoop 3.x high availability: Multiple standby NameNodes reduce metadata failover risk, while erasure coding can reduce storage overhead compared with three full replicas.

  • Data locality: Map tasks are scheduled near their HDFS inputs. Reducers are also scheduled near map-output data when cluster resources permit.

HDFS/YARN concepts remain relevant even where teams now use cloud object storage, Spark, or cloud warehouses because they illustrate distributed storage, fault tolerance, data locality, and resource scheduling.

Common Pitfalls

  • Incorrect partitioner: Splitting identical keys across reducers prevents correct aggregation.

  • Skew: A popular key can overload one reducer; salting or a two-stage aggregation can help.

  • Stateful reducers: Do not assume reducer calls occur in a particular order. Keep aggregation logic order-independent.

  • Unsafe combiners: A combiner must preserve the final aggregation semantics.

  • Small files: Many tiny HDFS files create excessive metadata and scheduling overhead; package them into larger files.

  • Memory pressure: Large partitions can exhaust reducer containers, so spills and bounded grouping strategies matter.

24

What is Hadoop’s SequenceFile format?

SequenceFile is Hadoop’s binary file format for storing serialized key-value records. It provides compact storage, class metadata, seeking, and splittable input for MapReduce, with the main trade-off that its records are harder to inspect and less efficient for analytical queries than columnar formats such as Parquet or ORC.

Read full answer

Applies toApache Hadoop 3.x

How It Works

A SequenceFile consists of a header followed by serialized key-value records.

  • Header: Contains a magic marker, format version, key and value class names, compression information, and optional file metadata.

  • Records: Contain encoded keys and values with framing that allows readers to identify record boundaries. The key and value types are identified by the class names in the header.

  • Sync markers: Periodic markers identify valid synchronization points. Readers use them to seek and to split SequenceFiles at suitable boundaries across HDFS blocks.

  • Compression: SequenceFiles can be uncompressed, record-compressed, or block-compressed. Block compression is useful for large files, although incompressible data may benefit little.

  • Java APIs: SequenceFile.Writer creates SequenceFiles, while SequenceFile.Reader iterates over records or seeks to a specified position.

When stored on HDFS, a SequenceFile benefits from HDFS durability and fault tolerance. The NameNode stores namespace metadata, while DataNodes hold the actual blocks. Hadoop’s default HDFS block size is 128 MB, and the default replication factor is 3, although clusters can override both settings.

Hadoop 3.x also supports erasure coding as an alternative to replication for suitable HDFS data and supports high-availability configurations with multiple standby NameNodes.

Hadoop and MapReduce Context

SequenceFiles were commonly used for intermediate MapReduce data:

  1. The mapper converts each input word into a (word, 1) record.

  2. The partitioner determines which reducer receives each key.

  3. An optional combiner performs local partial aggregation before the shuffle.

  4. The framework sorts and groups records by key, and the reducer aggregates the values.

MapReduce schedules tasks close to their input blocks for data locality. If a task fails, Hadoop can re-execute it, while HDFS block replication or erasure coding protects the underlying data.

Hadoop 3.x runs MapReduce applications through YARN:

  • The ResourceManager allocates cluster resources.

  • NodeManagers manage resources and execute tasks on individual workers.

  • Each application has an ApplicationMaster that coordinates its tasks.

SequenceFiles remain available in Hadoop 3.x, but many workloads now use Spark, which often runs on YARN and reads HDFS because it supports faster in-memory and iterative processing. Cloud object stores such as S3, GCS, and ADLS also provide alternatives to HDFS, although distributed storage, data locality, and fault tolerance remain important Hadoop interview topics.

Format Comparison

Format Structure Main Strength Main Weakness Typical Use
TextFile Human-readable lines Easy to inspect and process with standard tools High overhead and no native class metadata Logs and simple interchange
SequenceFile Binary key-value records Class-aware, splittable, and designed for Hadoop processing Not human-readable and has limited schema evolution Legacy MapReduce and Hadoop-native data
Parquet or ORC Columnar binary records Efficient compression and analytical queries Less natural for generic key-value processing Data warehouses and large analytical datasets

Common Pitfalls

  • Treating it as readable text: Use Hadoop-aware tools such as hdfs dfs -text or the SequenceFile.Reader API.

  • Confusing it with a columnar format: SequenceFile stores complete records sequentially; Parquet and ORC are generally better for analytical scans.

  • Ignoring schema compatibility: Readers need compatible key and value classes on their classpath. Avro or another schema-based format is usually preferable when records must evolve.

  • Creating many small files: Excessive file counts increase HDFS metadata pressure and YARN scheduling overhead, so compact appropriately sized files where possible.

  • Assuming fault tolerance comes from SequenceFile: Durability comes from HDFS replication or erasure coding, while task re-execution is provided by Hadoop’s fault-tolerant processing layer.

25

How do you optimize a Hadoop MapReduce job?

MapReduce optimization improves parallelism, reduces data movement and skew, and uses YARN resources efficiently without weakening fault tolerance. It addresses slow jobs by tuning the input layout, mapper and reducer processing, shuffle, partitioning, and resource allocation. The main trade-off is that aggressive tuning can increase memory pressure, duplicate work, or make correctness harder to reason about.

Read full answer

Applies toHadoop 3.x with Hadoop Streaming and Python 3.12

MapReduce Execution

A typical job follows these stages:

  1. HDFS input is divided into input splits, normally aligned with the 128 MB HDFS block size.

  2. Each mapper processes one split, potentially on the node containing that data to exploit data locality.

  3. Mapper output is partitioned, spilled to disk, and sorted.

  4. The shuffle phase transfers records to the nodes running reducers.

  5. Sorted partitions are merged, and reducers aggregate their assigned keys.

  6. Reducers write output files back to HDFS.

YARN provides this execution environment: the ResourceManager allocates resources, NodeManagers manage containers, and the ApplicationMaster coordinates the distributed job.

High-Impact Optimizations

Area Optimization Benefit and caution
Input design Combine small files into larger, efficiently splittable inputs Reduces mapper overhead; HDFS’s small-files problem affects NameNode metadata and scheduling
Mapper Aggregate partial results locally Reduces shuffle volume; only use a combiner when aggregation is associative and does not change results
Partitioning Use a suitable key and custom Partitioner when necessary Balances reducers; distribute hot keys carefully, often using salting
Sorting Use secondary sorting only when required Enables grouping by multiple fields but adds processing and comparator complexity
Reducers Choose a reducer count proportional to input volume and available memory Too few reducers cause bottlenecks; too many create tiny output files and scheduling overhead
Serialization Use compact binary formats such as SequenceFile where appropriate Reduces I/O and parsing overhead; plain text is useful for interoperability
Resource tuning Adjust container memory, JVM heap, and parallelism based on measured usage More memory can reduce spills, but excessive allocation wastes YARN capacity
Shuffle Minimize unnecessary key-value traffic and avoid expensive reducers The shuffle and merge phases often dominate execution time

A combiner performs partial aggregation in the mapper process and can reduce shuffle volume. Hadoop may invoke it zero or more times, so it must be safe to aggregate partial results and must not change the final result.

Fault Tolerance and Scale

  • HDFS stores blocks on DataNodes, while the NameNode manages metadata. The default HDFS replication factor is 3; erasure-coding configurations can use a different durability strategy.

  • Failed map or reduce tasks are automatically re-executed. Speculative execution can help when a task runs unusually slowly, although it adds resource overhead.

  • NameNode high availability uses one or more standby NameNodes, reducing a single metadata-service failure point.

  • Excessive speculative execution, over-sized JVM heaps, or too many concurrent containers can create contention rather than improve throughput.

  • Many organizations now use Spark, cloud object stores, or cloud warehouses for general workloads. Nevertheless, HDFS, YARN, data locality, shuffle behavior, and fault tolerance remain important Hadoop interview topics.

Code Example

The following Hadoop Streaming mapper emits each normalized word with a count of 1.

#!/usr/bin/env python3.12
import re
import sys

for line in sys.stdin:
    for word in re.findall(r"\b[\w']+\b", line.lower()):
        print(f"{word}\t1")

The reducer sums counts. It can also run as a combiner because summation is associative.

#!/usr/bin/env python3.12
import sys
from collections import defaultdict

totals = defaultdict(int)

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    totals[word] += int(count)

for word in sorted(totals):
    print(f"{word}\t{totals[word]}")

Prepare the scripts and submit the job. -files distributes them to YARN task containers; every worker node must provide python3.12.

set -euo pipefail

printf 'hadoop mapreduce hadoop\nmapreduce hadoop\n' > input.txt
chmod +x map.py reduce.py

hdfs dfs -mkdir -p /user/interview/input
hdfs dfs -put input.txt /user/interview/input/
hdfs dfs -ls -h /user/interview/input

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -name 'hadoop-streaming-*.jar' -print -quit)
test -n "$STREAMING_JAR"

hadoop jar "$STREAMING_JAR" \
  -files map.py,reduce.py \
  -input /user/interview/input \
  -output /user/interview/output \
  -mapper "python3.12 map.py" \
  -reducer "python3.12 reduce.py" \
  -combiner "python3.12 reduce.py" \
  -numReduceTasks 2

hdfs dfs -ls -h /user/interview/output
hdfs dfs -cat '/user/interview/output/part-*'
# Expected counts: hadoop 3, mapreduce 2

Measuring Performance

Use JobHistory, logs, and MapReduce counters rather than guessing. Important measurements include:

  • Mapper and reducer runtimes.

  • Bytes read, written, and shuffled.

  • Number of spills and merge operations.

  • Failed or retried tasks.

  • Records processed per mapper and reducer.

  • Reducer time skew caused by uneven partitions.

For token counting, mapper processing is approximately linear in input size, mapper sorting adds an O(m log m) upper bound for m records per mapper, and reducer-side merging requires roughly O(n log(p + 1)) comparisons across the job for n records and p mapper tasks. Exact costs also depend on spills, compression, skew, and reducer count.

Common Pitfalls

  • Using a combiner incorrectly: It can change results when aggregation is not associative.

  • Ignoring skew: A single popular key may overload one reducer despite an adequate reducer count.

  • Creating too many reducers: This produces many small output files and increases ApplicationMaster or task-scheduling overhead.

  • Treating every file as an input split: Non-splittable compression and tiny files limit parallelism.

  • Increasing heap blindly: Larger containers can cause spills, node exhaustion, or worse total cluster throughput.

  • Optimizing without measurements: Use counters and representative data before changing memory, reducers, or parallelism.

Interviewers commonly ask how you detect skew, why a combiner is not always safe, how data locality reduces network traffic, and how you distinguish a slow task from an overloaded reducer.

26

How do you diagnose and troubleshoot Hadoop performance issues?

Hadoop performance troubleshooting is a layered process that identifies bottlenecks in data locality, storage, scheduling, computation, shuffle, or networking. It explains where time and resources are consumed so the smallest safe corrective action can be applied. The main trade-off is fast diagnosis without adding enough diagnostic load to worsen an incident or weakening fault tolerance.

Read full answer

Applies toHadoop 3.x with Python 3.12 for Hadoop Streaming examples

Diagnostic Framework

  1. Establish the symptom and baseline: Determine whether the symptom is latency, low throughput, resource exhaustion, failed tasks, or uneven progress. Compare against a similar successful job, including task-duration distributions, counters, queue placement, and container metrics.

  2. Check HDFS: Use hdfs dfsadmin -report, hdfs fsck, hdfs dfs -du, and DataNode logs to investigate missing blocks, slow disks, replication delays, and metadata pressure.

  3. Check YARN: Inspect ResourceManager and NodeManager metrics for queue contention, pending containers, insufficient CPU or memory, disk pressure, and excessive container restarts.

  4. Check computation: Separate map, shuffle, reduce, GC, and spill time. For Spark, inspect executor pressure, partitions, shuffle size, and task skew.

  5. Change one variable at a time: Apply one controlled change, repeat the comparable workload, and compare latency, throughput, resource use, and failure rates.

HDFS, YARN, and MapReduce

  • NameNode and DataNode: The NameNode stores filesystem metadata; DataNodes store HDFS blocks. Common Hadoop 3.x defaults are a 128 MB block size and replication factor 3. Small files create disproportionate metadata and scheduling overhead, so compact, sequence, archive, or move cold data to object storage where appropriate.

  • NameNode HA: Hadoop 3.x can use multiple standby NameNodes with a shared edits journal and automatic failover. Verify JournalNode quorum, ZooKeeper failover controller, fencing, and standby state; an apparently idle standby may not be correctly tailing edits.

  • YARN: The ResourceManager allocates cluster resources, NodeManagers manage containers and node resources, and each ApplicationMaster requests containers and coordinates one application.

  • Fault tolerance: HDFS heals missing replicas, while YARN and MapReduce re-execute failed tasks. Speculative execution can duplicate slow tasks, so inspect attempt counts, retry logs, and failures across nodes rather than assuming another retry will solve the problem.

  • MapReduce flow: A mapper emits word<TAB>1; shuffle moves, groups, and sorts map output; reducers sum equal words. An optional combiner pre-aggregates locally, while the partitioner routes each key to a reducer.

  • Skew and locality: One exceptionally slow reducer usually indicates a hot key or partition. YARN normally schedules containers near input blocks to preserve data locality and reduce network transfer.

Inspect the ResourceManager, DataNode, and JobHistory interfaces, plus task logs under the configured log.user directory. This is commonly $HADOOP_HOME/logs/userlogs/ in a default installation, but hadoop.log.dir and log.user determine the actual location.

Ecosystem and Current Role

Component Role Troubleshooting focus
Hive SQL on Hadoop using execution engines such as Tez or Spark Query plans, partition pruning, skew, stage metrics
HBase Distributed wide-column store on HDFS Region hotspots, compaction, block-cache pressure
Pig Legacy data-flow language Legacy pipeline behavior and maintenance
Sqoop Retired project for relational-data ingestion Existing legacy pipelines only
Oozie Hadoop workflow scheduling Dependencies, retries, scheduler delays
ZooKeeper Distributed coordination Quorum, sessions, connection latency
Spark Faster in-memory engine often running on YARN and reading HDFS Shuffles, partitioning, executors, skew

Many teams now use S3, GCS, ADLS, Spark, and cloud warehouses, but HDFS/YARN concepts remain important interview topics because they illustrate distributed storage, locality, replication, and fault tolerance.

Code Example

Save these as mapper.py and reducer.py; Python 3.12 must be available on task hosts.

import sys

# Emit each token as a tab-separated key/value pair.
for line in sys.stdin:
    for word in line.split():
        sys.stdout.write(f"{word}\t1\n")
import sys

# Hadoop Streaming sorts reducer input by key.
current_word = None
total = 0

for line in sys.stdin:
    word, count_text = line.rstrip("\n").split("\t", 1)
    count = int(count_text)

    if current_word is None:
        current_word = word
        total = count
    elif word == current_word:
        total += count
    else:
        sys.stdout.write(f"{current_word}\t{total}\n")
        current_word = word
        total = count

if current_word is not None:
    sys.stdout.write(f"{current_word}\t{total}\n")
# Create deterministic local input.
printf 'a b a\nb c\n' > input.txt

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -name 'hadoop-streaming-*.jar' -print -quit)

hdfs dfs -mkdir -p "/user/$USER/wordcount"
hdfs dfs -put -f input.txt "/user/$USER/wordcount/input.txt"
hdfs dfs -ls -h "/user/$USER/wordcount"

# Remove any previous output so the job can be rerun.
hdfs dfs -rm -r -f "/user/$USER/wordcount/output"

hadoop jar "$STREAMING_JAR" \
  -input "/user/$USER/wordcount/input.txt" \
  -output "/user/$USER/wordcount/output" \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -files mapper.py,reducer.py \
  -numReduceTasks 1

hdfs dfs -cat "/user/$USER/wordcount/output/part-r-*"
# Final records: a 2, b 2, c 1

Common Pitfalls

  • Assuming CPU is the bottleneck: Check disk, network, shuffle, GC, and container allocation first.

  • Ignoring small files: A job can be dominated by metadata operations even when aggregate throughput looks healthy.

  • Tuning without a baseline: Record task distributions and counters before changing parameters.

  • Overlooking skew: Compare median and maximum task duration and inspect hot keys or partitions.

  • Reducing fault tolerance for speed: Erasure coding lowers storage overhead but has different reconstruction costs; do not remove protection without understanding those trade-offs.

  • Assuming HDFS is always required: Prefer the platform matching the workload while preserving the underlying locality and fault-tolerance reasoning.

  • Likely follow-up questions: Which metric identified the bottleneck, how was the change validated, and how would the fix behave under increased skew or node failure?

27

What is the role of a combiner in Hadoop MapReduce?

A combiner is an optional mini-reducer that aggregates mapper output before the data is shuffled across the network. It reduces intermediate data volume and improves performance, but it cannot replace the reducer because it sees only output produced locally by a mapper. Its main trade-off is cheaper shuffle and I/O at the cost of best-effort, partial aggregation whose execution may be repeated or skipped.

Read full answer

Applies toHadoop 3.x MapReduce

How It Works

In a typical MapReduce job:

  1. Map: Processes input splits and emits key-value pairs.

  2. Local combine: Optionally aggregates multiple values for the same key within each mapper’s output.

  3. Shuffle and sort: Partitions, transfers, and groups the intermediate data by key for the reducers.

  4. Reduce: Aggregates all values associated with each key and produces the authoritative final output.

The relevant components differ as follows:

Component Responsibility Key consideration
Mapper Generates intermediate key-value pairs Runs close to the input data
Combiner Performs local, partial aggregation Optional and not guaranteed to run
Partitioner Selects the destination reducer for each key Controls data distribution between reducers
Reducer Aggregates all values associated with each key across map tasks Produces the authoritative final output

A combiner generally receives the same key-value format expected by the reducer. It is safe only when the aggregation is associative and commutative, so combining values in different groups or orders produces the same final result. The framework may invoke a combiner multiple times on subsets of a mapper’s output, so the reducer must remain able to combine partial results.

Word Count Example

For word count, the mapper emits each word with a count of 1, and the reducer sums all counts associated with each word. A combiner can perform the same summation locally within each mapper’s output because addition is associative and commutative. This reduces the amount of repeated word data sent through the shuffle without changing the final counts.

A combiner cannot compute a global average correctly by averaging mapper-local averages because the mappers may have processed different numbers of records. The correct approach is to combine and sum the underlying counts and totals, then divide at the reducer.

Hadoop Context

  • HDFS: The NameNode stores namespace metadata and block locations, while DataNodes store the actual blocks. Hadoop 3.x uses a default block size of 128 MB and a typical replication factor of 3.

  • YARN: The ResourceManager allocates resources, NodeManagers manage containers and local resources, and an ApplicationMaster coordinates each application.

  • MapReduce: Executes distributed map and reduce tasks. Data locality moves computation to nodes near its input, while block replication and task re-execution provide fault tolerance.

  • Ecosystem: Hive provides SQL over Hadoop data, HBase is a wide-column store, Pig is a legacy data-flow tool, Sqoop is a retired Apache project, Oozie manages workflows, and ZooKeeper provides coordination. Spark is often faster for iterative or in-memory workloads and commonly runs on YARN while reading HDFS.

Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability. Many organizations now use cloud object storage and managed Spark or warehouse services, but HDFS, YARN, data locality, and fault-tolerance concepts remain important in interviews.

Best Practices

  • Use a combiner only when its aggregation is associative and commutative.

  • Keep mapper, combiner, and reducer key-value formats compatible.

  • Measure intermediate shuffle size before and after enabling a combiner; some jobs gain little because mapper output is already small or poorly grouped.

  • Avoid overly small files because they create excessive task overhead and increase NameNode metadata pressure.

  • Preserve task-retry safety because combiner execution may occur multiple times; failed attempts must not corrupt the final result.

  • Use a reducer to perform any final calculation that requires global knowledge, such as a global average or percentage.

Common Pitfalls

  • Assuming one combiner invocation receives all of a mapper’s output; it may receive only a subset, run multiple times, or be skipped.

  • Treating combiner output as the final, globally aggregated result.

  • Averaging partial averages without also aggregating the underlying counts and totals.

  • Using a combiner for an operation that is not associative or commutative and thereby changing the job’s result.

  • Assuming a combiner always reduces shuffle cost; its benefit depends on how well values are grouped and how much output it eliminates.

  • Follow-up questions: “Why is a combiner not guaranteed to run?”, “Which operations are safe for a combiner?”, “How would you compute an average correctly?”, and “What happens if a map task is retried?”

28

How can HDFS performance be optimized?

HDFS performance optimization means tuning data layout, block placement, replication, hardware utilization, and NameNode architecture so large-scale reads and writes complete efficiently. It addresses excessive metadata, small files, skewed data, remote I/O, and overloaded nodes. The main trade-off is that more parallelism or redundancy can increase storage, network, and recovery costs.

Read full answer

Applies toApache Hadoop 3.x (HDFS; Python 3.12 examples)

HDFS, YARN, and MapReduce

HDFS divides files into typically large blocks. The NameNode stores namespace and block metadata; the DataNodes store blocks and serve clients. Defaults include a 128 MB block size, replication factor 3, and rack awareness, with replicas spread across racks. In YARN, the ResourceManager allocates cluster resources, NodeManagers manage each node’s resources and containers, and an ApplicationMaster coordinates each application.

For the word-count example:

  1. Map: Input splits go to map tasks, which emit each word with count 1.

  2. Shuffle and sort: Map outputs move to reducers; keys are partitioned and sorted so values for a word arrive together.

  3. Reduce: Each reducer sums the counts for its words.

  4. Combiner: Optional local aggregation reduces shuffled data, but it can change output when key/value semantics are order-sensitive.

  5. Partitioner: The default hash partitioner assigns each key to a reducer, spreading words across reducers.

  6. Locality: YARN prefers scheduling containers on nodes holding input blocks, avoiding remote reads.

Key Optimization Techniques

  • Manage small files and use efficient formats: Many small files create excessive NameNode metadata and client RPCs. Combine records before writing; use Parquet or ORC and compression to reduce I/O, although compression adds CPU cost.

  • Tune block size: Larger blocks reduce metadata and connection overhead and improve sequential throughput, but reduce the number of independent blocks available for parallel processing and increase the time required to fill a block.

  • Balance and preserve locality: Use hdfs balancer to redistribute unevenly placed blocks. Schedule MapReduce containers near their input blocks; remote reads consume network bandwidth.

  • Reduce replication judiciously: Replication factor 3 is a durability default, not optimal for every dataset. Lower replication reduces storage and write-pipeline costs but decreases fault tolerance.

  • Maintain healthy nodes and control traffic: Replace failed disks and monitor capacity, bad blocks, disk throughput, RPC latency, garbage collection, and network saturation. Avoid excessive concurrent writers against one NameNode endpoint.

Fault Tolerance and High Availability

HDFS tolerates failures through block replication and task re-execution. If a DataNode fails, another replica remains available; MapReduce reruns tasks that lose inputs or outputs.

For critical deployments, configure NameNode high availability with an active and standby NameNode backed by JournalNodes. Hadoop 3.x supports multiple standby NameNodes while retaining one active metadata writer.

Hadoop 3.x also supports erasure coding, which can store large datasets with less overhead than three-way replication. It trades additional CPU and network work during reconstruction for lower storage cost, making it attractive for cold or archival data. Frequently accessed data often benefits more from replication because reads are simpler and faster.

Code Example

The following Hadoop Streaming example counts words in an HDFS file. The mapper emits tab-separated pairs, and the reducer sums the values after the framework groups them by key.

mapper.py:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

reducer.py:

#!/usr/bin/env python3
import sys

current_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    value = int(value)

    if current_word is None:
        current_word = word

    if word == current_word:
        count += value
    else:
        print(f"{current_word}\t{count}")
        current_word = word
        count = value

if current_word is not None:
    print(f"{current_word}\t{count}")

Test the scripts locally with deterministic input:

cat > input.txt <<'EOF'
hadoop hdfs
hadoop mapreduce
EOF

cat input.txt | python3 mapper.py | LC_ALL=C sort | python3 reducer.py

Expected output:

hadoop	2
hdfs	1
mapreduce	1

Upload and inspect the file:

hdfs dfs -mkdir -p "/user/$USER/wordcount/input"
hdfs dfs -put input.txt "/user/$USER/wordcount/input/"
hdfs dfs -ls -h "/user/$USER/wordcount/input"

Run the MapReduce job. The -files option distributes the scripts to the cluster, and Python 3.12 must be available on the worker nodes:

STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/mapreduce" \
  -name 'hadoop-streaming-*.jar' -print -quit)"

hadoop jar "$STREAMING_JAR" \
  -input "/user/$USER/wordcount/input" \
  -output "/user/$USER/wordcount/output" \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py"

Common Pitfalls

  • Increasing block size without considering parallelism and recovery time

  • Treating more replication as universally faster

  • Assuming hdfs balancer solves the small-file problem

  • Scheduling tasks away from their HDFS blocks

  • Optimizing NameNode memory without measuring RPC latency and garbage collection

  • Ignoring cloud object storage’s role: many teams now use S3, GCS, or ADLS with Spark or cloud warehouses, while retaining HDFS concepts such as distributed storage, fault tolerance, and data locality in interviews

  • Failing to monitor skew, failed disks, RPC queues, network traffic, and DataNode utilization together

29

How do you optimize Hadoop job scheduling on YARN?

YARN job scheduling optimization allocates containers and guides task placement so distributed work finishes efficiently without starving other tenants. It addresses resource contention, data movement, skew, and slow tasks. The main trade-off is maximizing utilization and locality while preserving fairness, controlling memory pressure, and limiting scheduling overhead.

Read full answer

Applies toApache Hadoop 3.x (HDFS, YARN, and MapReduce; Python 3.12 examples)

YARN Scheduling Model

  • ResourceManager: Allocates resources among applications and queues through a pluggable scheduler. It does not directly place individual MapReduce tasks.

  • Schedulers: CapacityScheduler provides hierarchical queues and capacity guarantees; FairScheduler provides weighted, flexible sharing.

  • NodeManager: Manages containers and local resources on each worker node.

  • ApplicationMaster: Requests containers, coordinates tasks, and monitors an application. Each MapReduce application has its own ApplicationMaster.

  • HDFS and fault tolerance: The NameNode stores metadata, while DataNodes store blocks. Hadoop 3.x defaults include a 128 MB block size and replication factor of 3. MapReduce re-executes failed tasks, using locality when possible.

  • Processing flow: A MapReduce application performs mapping, shuffle and sort, and reduction. Its locality-aware ApplicationMaster prefers node-local or rack-local containers, while the YARN scheduler allocates the requested resources.

Optimization Priorities

Scheduler How it works Strengths Weaknesses When to use Typical scale
CapacityScheduler Uses hierarchical queues with guaranteed capacity and optional preemption Predictable capacity and tenant isolation May reserve capacity a tenant does not consume Organizations requiring queue-level guarantees Large multi-tenant clusters
FairScheduler Shares resources among application or user queues using weights and minimum shares Flexible sharing and good utilization for variable demand Provides less predictable hard capacity isolation Clusters with changing workload mixes Large multi-tenant clusters
  • Right-size containers: Request realistic memory and vcores. Excessive requests strand capacity, while insufficient heap can cause retries or excessive garbage collection.

  • Preserve locality: Have the MapReduce ApplicationMaster request containers near HDFS inputs and map outputs. This reduces remote reads, disk I/O, and network traffic.

  • Reduce skew: Use suitable partitioners and monitor reducer metrics. Carefully salt skewed composite keys when the aggregation semantics permit it.

  • Use combiners appropriately: A combiner reduces shuffle volume, but its operation must be associative and commutative and must not depend on global ordering.

  • Control speculation: Speculative execution can recover from slow tasks, but it consumes extra resources and may duplicate work.

  • Control small files and monitor: Consolidate small inputs and track queue utilization, retries, shuffle volume, locality, and disk or network bottlenecks.

Ecosystem and Platform Trade-offs

Hive provides SQL access, HBase is a wide-column store, Pig is a legacy batch tool, Sqoop is a retired Apache project, Oozie handles workflows, and ZooKeeper provides coordination. Spark is an in-memory engine that commonly runs on YARN and reads HDFS, often replacing MapReduce for iterative or interactive workloads.

Hadoop 3.x adds erasure coding and multiple standby NameNodes. Erasure coding reduces storage overhead but adds reconstruction work, while replicated blocks remain convenient for frequently read data. Many organizations now use cloud object storage, cloud warehouses, or Spark outside YARN, but HDFS/YARN concepts such as data locality, distributed storage, and fault tolerance remain relevant.

Code Example

This Hadoop Streaming example performs word counting with Python 3.12. Provision python3.12 in the YARN container PATH; -files distributes the scripts to worker containers.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.lower().split():
        print(f"{word}\t1")
# reducer.py
import sys

current_word = None
count = 0

# Hadoop Streaming sorts reducer input by key before invoking this script.
for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)

    if current_word is None:
        current_word = word
        count = 0

    if word == current_word:
        count += int(value)
    else:
        print(f"{current_word}\t{count}")
        current_word = word
        count = int(value)

if current_word is not None:
    print(f"{current_word}\t{count}")
# Run from the directory containing mapper.py and reducer.py.
printf 'Hadoop runs MapReduce\nHadoop schedules tasks\n' > words.txt

hdfs dfs -mkdir -p "/user/$USER/in"
hdfs dfs -put words.txt "/user/$USER/in/words.txt"
hdfs dfs -ls "/user/$USER/in"

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/mapreduce" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' -print -quit)

OUTPUT="/user/$USER/wordcount-output-$(date +%s)-$$"

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -input "/user/$USER/in/words.txt" \
  -output "$OUTPUT" \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 1

hdfs dfs -ls "$OUTPUT"
hdfs dfs -cat "$OUTPUT/part-*"

For the supplied input, hadoop occurs twice and every other word occurs once.

Best Practices

  • Establish a baseline using queue utilization, completion time, shuffle traffic, locality, and failure rates.

  • Change one scheduling or resource parameter at a time and test against representative data and skew.

  • Keep retry behavior and speculative execution conservative in shared clusters.

  • Ensure the container runtime has Python 3.12 and all required dependencies.

Common Pitfalls

  • Requesting too many vcores or too much heap for every container.

  • Assuming more concurrent applications always improves throughput.

  • Ignoring reducer skew while blaming the scheduler.

  • Creating many tiny HDFS files.

  • Enabling speculation broadly and wasting resources.

  • Ignoring remote reads caused by poor data placement.

Common follow-up questions include which scheduler suits a multi-tenant cluster, how to diagnose one slow reducer, and how data locality affects network usage and task runtime.

30

How do you manage memory and CPU resources in Hadoop 3.x?

Effective YARN resource management balances container size and concurrency against hardware utilization, network traffic, and failure impact. Larger containers can reduce spilling but permit fewer tasks to run concurrently, while aggressive CPU allocation can cause contention. Resource settings should therefore be derived from workload measurements rather than fixed assumptions.

Read full answer

Applies toApache Hadoop 3.x with YARN and MapReduce

Resource Model

HDFS separates storage from computation. The NameNode manages namespace metadata, while DataNodes store the default 128 MB blocks with a default replication factor of 3.

  • ResourceManager: Coordinates cluster-wide resources and delegates application placement to a configured scheduler.

  • NodeManager: Reports and manages resources on one worker node.

  • ApplicationMaster: Coordinates the tasks or executors belonging to one application.

  • Container: The allocation unit for a process; memory is requested in MB and CPU in virtual cores, or vcores.

Memory Management

  • Set mapreduce.map.memory.mb and mapreduce.reduce.memory.mb from measured working-set requirements rather than applying one size to every task.

  • Keep the JVM heap below the container limit. Configure heap through mapreduce.map.java.opts and mapreduce.reduce.java.opts, while reserving memory for direct buffers, decompression, native libraries, and operating-system overhead.

  • Monitor spill volume, garbage-collection time, resident memory, and failed containers. Large or persistent spills often indicate insufficient memory or unexpectedly large records.

  • Avoid oversized containers: they reduce task concurrency, increase failure impact, and can lengthen garbage-collection pauses.

  • For Spark on YARN, size driver memory, executor memory, and spark.executor.memoryOverhead separately. Dynamic allocation can improve utilization, but many tiny executors also create scheduling overhead.

CPU and Scheduling

  • Use the Capacity Scheduler or Fair Scheduler to isolate teams and workloads with queues, capacity limits, and placement policies.

  • Configure yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores to describe resources available to YARN on each node.

  • Request CPU with mapreduce.map.cpu.vcores and mapreduce.reduce.cpu.vcores. A vcore is primarily a scheduling unit; hard CPU isolation requires suitable node-level controls such as cgroups.

  • Do not oversubscribe CPU. Excess concurrency causes context switching and contention without necessarily increasing total throughput.

  • Use speculative execution conservatively because it can recover a genuinely slow task, but it also duplicates CPU, memory, and I/O consumption.

Locality, Fault Tolerance, and Data Pipeline

  • HDFS replication protects against DataNode failures, while YARN re-executes failed tasks. Schedule tasks near their input blocks to reduce network traffic, while accounting for skewed data.

  • In MapReduce word count, the map step emits each word and 1. The framework partitions map output, transfers it during shuffle, sorts records by key, and reducers aggregate the counts.

  • A combiner performs partial aggregation near the mapper before shuffle. A partitioner decides which reducer receives each key; poorly chosen partitioners can overload one reducer.

  • Deploy NameNode high availability with one active NameNode, one or more standbys in Hadoop 3.x, JournalNodes for shared edits, and automatic failover with proper fencing. Only the active NameNode serves client requests.

  • Hadoop 3.x also supports erasure coding, which reduces storage overhead at the cost of reconstruction CPU and network activity.

  • Consolidate small files because they create excessive tasks, metadata operations, and inefficient I/O.

  • Many teams now use cloud object storage, cloud warehouses, or Spark instead of HDFS and YARN, but distributed resource allocation, data locality, spill control, and fault tolerance remain relevant.

Code Example

The following Hadoop Streaming word count uses Python 3.12. The mapper emits tab-separated word counts, and the reducer aggregates them.

mapper.py:

#!/usr/bin/env python3.12
import sys

for line in sys.stdin:
    for word in line.lower().split():
        print(f"{word}\t1")

reducer.py:

#!/usr/bin/env python3.12
import sys
from collections import defaultdict

counts = defaultdict(int)

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t")
    counts[word] += int(count)

for word, count in counts.items():
    print(f"{word}\t{count}")

Run these commands from the directory containing both scripts. The python3.12 executable must be available on every NodeManager that runs the job.

printf 'hadoop hdfs\nhadoop mapreduce\n' > input.txt

hdfs dfs -mkdir -p "/user/$USER/wordcount/input"
hdfs dfs -put input.txt "/user/$USER/wordcount/input/"
hdfs dfs -ls -h "/user/$USER/wordcount/input"

STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -name 'hadoop-streaming-*.jar' | head -n 1)"

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -input "/user/$USER/wordcount/input" \
  -output "/user/$USER/wordcount/output" \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 2

hdfs dfs -cat "/user/$USER/wordcount/output/part-*"

The deterministic counts are hadoop 2 and mapreduce 2, although reducer-file order may vary. The output directory must not already exist.

Common Pitfalls

  • Using one configuration for every workload: Map, reduce, and Spark tasks have different profiles.

  • Measuring only JVM heap: Native and off-heap memory also consume the container allowance.

  • Treating vcores as guaranteed physical cores: Scheduling units do not automatically provide strict CPU isolation.

  • Ignoring skew or spills: Evenly sized containers can still leave one reducer overloaded.

  • Creating many small files: Consolidate inputs where practical.

  • Enabling speculation without measurement: It can waste resources and duplicate expensive work.

Interviewers commonly ask how to distinguish memory pressure from CPU starvation, diagnose a slow reducer, size containers, and explain how YARN allocation affects locality and fault recovery.

31

How do you write a Java MapReduce word-count job?

MapReduce is Hadoop’s distributed batch-processing model: mappers produce key-value pairs, the framework groups and sorts them by key, and reducers aggregate each group. A Java word-count job provides fault-tolerant processing with data locality, although it is more verbose and generally slower than in-memory engines such as Spark.

Read full answer

Applies toApache Hadoop 3.x with Java 11; Python 3.12 for the Hadoop Streaming example

MapReduce Execution Flow

For the input hadoop hadoop mapreduce, a typical job proceeds as follows:

  1. Map: Each input line is tokenized. The mapper emits hadoop → 1, hadoop → 1, and mapreduce → 1.

  2. Partition and combine: The default HashPartitioner selects a reducer for each key. An optional combiner performs map-side preaggregation when its operation is associative and commutative.

  3. Shuffle and sort: Map output is transferred to the appropriate reducers and sorted by key.

  4. Reduce: The reducer sums the values for each key, producing hadoop → 2 and mapreduce → 1.

  5. Output: TextOutputFormat writes the results, normally as one or more part-* files.

Hadoop Architecture and Fault Tolerance

  • HDFS: The NameNode stores metadata, while DataNodes store data blocks. The default block size is 128 MiB and the default replication factor is 3.

  • YARN: The ResourceManager allocates resources, each NodeManager manages a worker, and the application’s ApplicationMaster coordinates its MapReduce tasks.

  • Fault tolerance: A failed task can be re-executed, and HDFS continues serving replicated blocks when a DataNode fails.

  • Data locality: Map tasks are scheduled near their input blocks, and reducers consume map output locally where possible to reduce network transfers.

  • High availability: Hadoop 3.x supports erasure coding and an active NameNode with multiple standby NameNodes, although replication remains the default.

  • Ecosystem and current role: Hive provides SQL on Hadoop, HBase is a wide-column store, Pig is legacy, Sqoop is retired, Oozie handles workflows, ZooKeeper coordinates services, and Spark is an in-memory engine often running on YARN and reading HDFS. Many teams now use Spark, cloud object stores, or cloud warehouses, but HDFS and YARN concepts remain interview staples.

Code Example

The native Java implementation uses the reducer as an optional combiner. Save it as WordCountDriver.java:

import java.io.IOException;
import java.util.Locale;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public final class WordCountDriver extends Configured implements Tool {
    public static final class WordMapper
            extends Mapper<LongWritable, Text, Text, IntWritable> {
        private final IntWritable one = new IntWritable(1);
        private final Text word = new Text();

        @Override
        protected void map(LongWritable key, Text line, Context context)
                throws IOException, InterruptedException {
            // Normalize and split each line into word tokens.
            String normalized = line.toString().toLowerCase(Locale.ROOT);
            String[] tokens = normalized.split("[^\\p{L}\\p{N}]+");

            for (String token : tokens) {
                if (!token.isEmpty()) {
                    word.set(token);
                    context.write(word, one);
                }
            }
        }
    }

    public static final class WordReducer
            extends Reducer<Text, IntWritable, Text, IntWritable> {
        private final IntWritable total = new IntWritable();

        @Override
        protected void reduce(Text key, Iterable<IntWritable> values,
                              Context context)
                throws IOException, InterruptedException {
            int sum = 0;
            for (IntWritable value : values) {
                sum += value.get();
            }
            total.set(sum);
            context.write(key, total);
        }
    }

    @Override
    public int run(String[] args) throws Exception {
        if (args.length != 2) {
            throw new IllegalArgumentException(
                    "Usage: WordCountDriver <input-path> <output-path>");
        }

        Configuration configuration = getConf();
        Job job = Job.getInstance(configuration, "Word Count");
        job.setJarByClass(WordCountDriver.class);
        job.setMapperClass(WordMapper.class);
        job.setCombinerClass(WordReducer.class);
        job.setReducerClass(WordReducer.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        job.setNumReduceTasks(2);

        // Configure the HDFS input and output paths.
        TextInputFormat.addInputPath(job,
                new org.apache.hadoop.fs.Path(args[0]));
        TextOutputFormat.setOutputPath(
                job, new org.apache.hadoop.fs.Path(args[1]));

        return job.waitForCompletion(true) ? 0 : 1;
    }

    public static void main(String[] args) throws Exception {
        System.exit(ToolRunner.run(
                new Configuration(), new WordCountDriver(), args));
    }
}

The equivalent Hadoop Streaming mapper reads standard input and writes tab-separated pairs:

# mapper.py
import sys

for line in sys.stdin:
    for word in line.lower().split():
        print(f"{word}\t1")

The Streaming reducer consumes keys in sorted order and prints each final count:

# reducer.py
import sys

current_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    value = int(value)

    if current_word is not None and word != current_word:
        print(f"{current_word}\t{count}")
        count = 0

    current_word = word
    count += value

if current_word is not None:
    print(f"{current_word}\t{count}")

Build and Run on HDFS

Compile, package, create sample input, and upload it:

mkdir -p target/classes
javac --release 11 -cp "$(hadoop classpath)" \
  -d target/classes WordCountDriver.java
jar cf wordcount.jar -C target/classes .

printf '%s\n' 'hadoop hadoop mapreduce' > input.txt
hdfs dfs -mkdir -p "/user/$USER/input"
hdfs dfs -put input.txt "/user/$USER/input/"
hdfs dfs -ls -h "/user/$USER/input/"

Run the Java job and inspect its output:

hadoop jar wordcount.jar WordCountDriver \
  "/user/$USER/input" "/user/$USER/output"

hdfs dfs -ls -h "/user/$USER/output"
hdfs dfs -cat "/user/$USER/output/part-*" | sort -k2,2nr | head

For Streaming, distribute both scripts and use a separate output directory:

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools" \
  -name 'hadoop-streaming-*.jar' -print -quit)

hadoop jar "$STREAMING_JAR" \
  -input "/user/$USER/input" \
  -output "/user/$USER/output-streaming" \
  -files mapper.py,reducer.py \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -D mapreduce.job.reduces=1

Every worker must provide Python 3.12, and the output path must not already exist.

Common Pitfalls

  • Existing output path: Choose a new path or deliberately remove the previous output.

  • Inconsistent tokenization: Decide explicitly how punctuation, case, underscores, hyphens, and apostrophes affect word identity.

  • Small files: Many tiny files increase NameNode metadata pressure and mapper overhead; combine or compact them where possible.

  • Unsafe combiner or partitioning assumptions: A combiner must preserve aggregation semantics, and parallel reducers do not provide one globally ordered output.

  • Choosing the wrong engine: Spark or a cloud warehouse is often better for iterative analytics or interactive queries, while MapReduce remains suitable for large fault-tolerant batch jobs.

32

How do I create a script that copies local files to HDFS?

Use hdfs dfs -put in a shell script to transfer files from the local file system into an HDFS directory. The HDFS upload stores files as one or more blocks, writes them to DataNodes, and records their metadata in the NameNode; the default block size is 128 MB and the default replication factor is 3. The main trade-off is straightforward operational tooling versus the need for correct client configuration, permissions, and awareness that using -f can overwrite files with the same name.

Read full answer

Applies toApache Hadoop 3.x (HDFS client and Bash)

How the Upload Works

  • The hdfs client reads the local file and divides it into HDFS blocks.

  • The client requests block allocation and DataNode locations from the NameNode, which stores metadata rather than file contents.

  • Blocks are written directly to the DataNodes and replicated for fault tolerance.

  • This script intentionally requires an absolute path on the cluster’s default HDFS filesystem, such as /user/alice/incoming. The client can also accept fully qualified URIs such as hdfs://host:port/path, but this script does not.

  • Uploading a file does not launch a MapReduce job or consume YARN resources.

Hadoop Context

Component Role
HDFS Distributed storage for the uploaded files
YARN Resource management and job execution through the ResourceManager, NodeManagers, and ApplicationMaster
MapReduce Distributed batch processing model involving map, shuffle and sort, and reduce phases
Hive SQL-based analytics over data stored in Hadoop
HBase Distributed wide-column NoSQL database
Pig Legacy data-flow language and execution platform
Sqoop Retired Apache project for moving relational data into Hadoop
Oozie Hadoop workflow and job scheduling
ZooKeeper Distributed coordination and synchronization
Spark Faster, primarily in-memory processing engine that commonly runs on YARN and reads HDFS

Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability, although erasure coding is not automatically equivalent to three-way replication.

Code Example

Save the following as copy_to_hdfs.sh:

#!/usr/bin/env bash
set -euo pipefail

usage() {
  printf 'Usage: %s <hdfs-directory> <local-file> [local-file ...]\n' "$0" >&2
  exit 2
}

# Require an HDFS destination and at least one local file.
(( $# >= 2 )) || usage

destination=$1
shift

# Requiring an absolute path avoids ambiguity with the local directory.
if [[ "$destination" != /* ]]; then
  printf 'HDFS destination must be an absolute path, such as /user/alice/incoming\n' >&2
  exit 2
fi

# Validate every source before modifying HDFS.
sources=()
for source in "$@"; do
  if [[ ! -f "$source" ]]; then
    printf 'Local file not found: %s\n' "$source" >&2
    exit 1
  fi

  # Convert relative paths to absolute paths for predictable uploads.
  sources+=("$(realpath -- "$source")")
done

# Normalize the destination while preserving the HDFS root.
destination="${destination%/}"
[[ -n "$destination" ]] || destination="/"

# Create the destination directory if it does not exist.
hdfs dfs -mkdir -p "$destination"

# A trailing slash makes HDFS preserve each source file's basename.
put_target="$destination"
if [[ "$put_target" != */ ]]; then
  put_target="$put_target/"
fi

for source in "${sources[@]}"; do
  hdfs dfs -put -f "$source" "$put_target"
done

# Show the uploaded files and their sizes.
hdfs dfs -ls -h "$destination"

Run it with:

chmod +x copy_to_hdfs.sh

./copy_to_hdfs.sh /user/alice/incoming ./report.csv ./orders.csv

Equivalent direct commands are:

hdfs dfs -mkdir -p /user/alice/incoming
hdfs dfs -put -f ./report.csv /user/alice/incoming/
hdfs dfs -put -f ./orders.csv /user/alice/incoming/
hdfs dfs -ls -h /user/alice/incoming

Operational Considerations

  • Client setup: The machine running the script needs hdfs on PATH, valid Hadoop client configuration, access to the NameNode, and filesystem permissions.

  • Overwrite behavior: hdfs dfs -put -f replaces an existing destination file with the same name. Omit -f when accidental replacement must be prevented.

  • Small files: HDFS performs poorly with many tiny files because each file consumes metadata and block resources. Store compact formats such as Parquet or combine records before uploading.

  • Naming conflicts: Files from different local directories with the same basename overwrite one another when uploaded into the same destination directory.

  • Cloud adoption: Many teams now use S3, GCS, or ADLS with cloud-native tools. The underlying ideas—distributed storage, replication, fault tolerance, and data locality—remain important, but hdfs dfs is specific to Hadoop-compatible HDFS clusters.

Common Pitfalls

  • Confusing a local path such as ./data with an HDFS path such as /user/alice/data.

  • Using -f without understanding that existing files may be overwritten.

  • Uploading many small files instead of compacting them first.

  • Assuming the NameNode stores file contents; it stores metadata, while DataNodes store the blocks.

  • Omitting set -euo pipefail, which can allow later uploads to continue after an earlier failure.

  • Expecting this command to use YARN or MapReduce; it is an HDFS client operation.

33

How do I summarize data in a Hive table using SQL?

Hive SQL summarizes a table by grouping rows and calculating aggregate measures such as COUNT, SUM, and AVG. Hive converts the query into an execution plan that reads the table’s files—commonly stored in HDFS—and runs through Hadoop’s execution layer. The main trade-off is simple, familiar SQL versus the computational and I/O cost of scanning, shuffling, and aggregating large datasets.

Read full answer

Applies toApache Hive 3.x with Hadoop 3.x

How the Query Works

Hive separates a table’s schema and location from its physical data:

  • The Metastore stores table definitions, column types, partitions, and storage locations.

  • Table data is stored as files in a warehouse directory or another HDFS-compatible location.

  • A query such as SELECT category, SUM(amount) ... GROUP BY category is parsed, optimized, and converted into one or more execution jobs.

  • Depending on the cluster configuration, those jobs can run through MapReduce or Tez, commonly managed by YARN.

  • The aggregation reduces many input rows to one result row per category.

Hadoop Execution Context

  • HDFS: The NameNode stores namespace and block metadata, while DataNodes hold replicated data blocks. Hadoop 3.x defaults to 128 MB blocks and a replication factor of 3. Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability.

  • YARN: The ResourceManager allocates cluster resources, NodeManagers manage individual workers, and an ApplicationMaster coordinates each distributed application.

  • MapReduce: The mapper processes records, the framework shuffles and sorts intermediate key-value pairs, and reducers aggregate them. A combiner performs local pre-aggregation, while a partitioner determines which reducer receives each key.

  • Fault tolerance: HDFS recovers from failed blocks using replication or erasure coding, and YARN can re-execute failed tasks. Scheduling near the input’s location reduces network transfer through data locality.

Code Example

The following is a self-contained Hive demonstration using a small orders table:

CREATE DATABASE IF NOT EXISTS interview_demo;
USE interview_demo;

DROP TABLE IF EXISTS orders;

CREATE TABLE orders (
    order_id BIGINT,
    customer_id BIGINT,
    category STRING,
    amount DOUBLE,
    order_date DATE
)
STORED AS ORC;

INSERT INTO TABLE orders VALUES (1001, 10, 'books', 25.50, '2024-01-05');
INSERT INTO TABLE orders VALUES (1002, 11, 'books', 40.00, '2024-01-07');
INSERT INTO TABLE orders VALUES (1003, 10, 'electronics', 120.00, '2024-01-08');
INSERT INTO TABLE orders VALUES (1004, 12, 'electronics', 80.00, '2024-01-09');
INSERT INTO TABLE orders VALUES (1005, 11, 'clothing', 35.00, '2024-01-10');

SELECT
    category,
    COUNT(*) AS order_count,
    SUM(amount) AS total_amount,
    ROUND(AVG(amount), 2) AS average_order_amount,
    MIN(order_date) AS first_order_date,
    MAX(order_date) AS latest_order_date
FROM orders
GROUP BY category
ORDER BY total_amount DESC;

A related Hadoop Streaming word count demonstrates the underlying map-shuffle-reduce model:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.lower().split():
        print(f"{word}\t1")
#!/usr/bin/env python3
import sys

counts = {}

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    counts[word] = counts.get(word, 0) + int(count)

for word, count in sorted(counts.items()):
    print(f"{word}\t{count}")

Place the scripts as mapper.py and reducer.py, then run:

printf 'hadoop hive mapreduce\nhadoop hive\n' > words.txt

hdfs dfs -mkdir -p /input
hdfs dfs -put -f words.txt /input/words.txt
hdfs dfs -ls -h /input

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' -print -quit)

OUT="/output/wordcount_$(date +%s)"

hadoop jar "$STREAMING_JAR" \
  -input /input/words.txt \
  -output "$OUT" \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -numReduceTasks 1

hdfs dfs -ls -h "$OUT"

Performance Guidance

  • Partition large tables by a frequently filtered column, such as order_date, and include partition predicates so Hive can prune irrelevant data.

  • Prefer ORC or Parquet for large analytical tables because they support compression and column pruning.

  • Avoid SELECT * when only a few columns are needed.

  • Use COUNT(*) to count rows, but COUNT(column) when non-null values should be counted.

  • Compact small files before processing them; excessive small files increase task scheduling and metadata overhead.

Common Pitfalls

  • A query without a partition filter may scan the entire table.

  • AVG ignores NULL values, so its result may differ from dividing SUM by COUNT(*).

  • Hive 3.x supports ACID transactions for eligible managed tables, but they have restrictions and do not provide unrestricted multi-statement rollback like a conventional relational database.

  • Common interview follow-ups ask how HDFS recovers blocks, how YARN differs from MapReduce, where the NameNode metadata lives, how partition pruning works, and why small files hurt performance.

34

How do I transform data with an Apache Pig Latin script?

Apache Pig Latin is a declarative language for batch data transformations on Hadoop. It solves large-scale filtering, grouping, aggregation, joining, and loading with concise data-flow statements, trading ecosystem currency and flexibility for familiar distributed batch processing.

Read full answer

Applies toApache Pig 0.17.0 on Hadoop 3.x; Python 3.12 for Hadoop Streaming examples

Using Pig Latin

A Pig script defines a logical plan using operators such as LOAD, FOREACH, GROUP, JOIN, FILTER, ORDER, and STORE. On Hadoop, Pig optimizes this plan and submits the required distributed jobs; local mode is intended only for development and testing.

Parameters can externalize values without editing the script. For example, $inputPath, $outputPath, and $minTotal are supplied with the pig -param name=value option.

Hadoop Execution and Fault Tolerance

HDFS stores blocks on DataNodes, while the NameNode manages metadata. Its default block size is 128 MB and its default replication factor is 3. YARN supplies the ResourceManager, NodeManagers, and a per-job ApplicationMaster, while scheduling tasks near the HDFS data to improve data locality.

For word count, MapReduce proceeds as follows:

  1. Map: Each input record becomes key-value pairs such as ("hadoop", 1).

  2. Combiner: Optional map-side aggregation reduces transferred data; it is safe only for associative and commutative operations.

  3. Partitioner: Selects the reducer responsible for each key.

  4. Shuffle and sort: Groups equal keys and orders their values.

  5. Reduce: Aggregates each key’s values into the final result.

Hadoop re-executes failed tasks and recreates missing replicas from surviving copies. Hadoop 3.x also supports erasure coding, which reduces storage overhead at the cost of additional reconstruction work, and multiple standby NameNodes for high availability.

Ecosystem Placement

Tool How it works Strengths Weaknesses When to use Typical scale
Apache Pig Compiles a declarative plan into distributed batch operations Concise ETL and aggregation Legacy ecosystem; less flexible than Spark or cloud SQL Existing Hadoop batch pipelines HDFS-scale batch jobs
Apache Hive Translates SQL-like queries into execution plans Familiar SQL and warehouse workflows Higher latency for some interactive workloads SQL analytics and reporting Large analytical warehouses
Apache Spark Runs DAGs with reusable in-memory transformations Broad, fast batch, streaming, and ML engine More operational and programming complexity New distributed data-processing workloads Large clusters and streaming workloads

HBase is a wide-column NoSQL store, Oozie schedules workflows, and ZooKeeper provides distributed coordination. Sqoop is a retired Apache project and should not be selected for new designs. Spark commonly runs on YARN and reads HDFS, while many modern systems use S3, GCS, ADLS, Spark, or cloud warehouses instead.

Code Example

This example creates a small CSV file, transforms it with Pig, and stores the result:

cat > events.csv <<'CSV'
u1,books,40
u1,books,25
u2,books,70
u2,electronics,20
u2,electronics,45
u1,electronics,10
CSV

cat > transform.pig <<'PIG'
events = LOAD '$inputPath' USING PigStorage(',')
         AS (user_id:chararray, category:chararray, amount:int);

grouped = GROUP events BY (user_id, category);

totals = FOREACH grouped GENERATE
         group.user_id AS user_id,
         group.category AS category,
         SUM(events.amount) AS total_amount,
         COUNT(*) AS transaction_count;

summaries = FILTER totals BY total_amount >= $minTotal;
ordered = ORDER summaries BY total_amount DESC, user_id ASC;

DUMP ordered;
STORE ordered INTO '$outputPath' USING PigStorage(',');
PIG

pig -x local \
    -param inputPath=events.csv \
    -param minTotal=50 \
    -param outputPath="summary-output-$$" \
    transform.pig

The deterministic DUMP order is (u2, books, 70), (u1, books, 65), and (u2, electronics, 65). The stored relation may be split across output part files.

The equivalent Hadoop Streaming word count uses these Python programs:

# mapper.py
import re
import sys

for line in sys.stdin:
    for word in re.findall(r"\w+", line.lower()):
        print(f"{word}\t1")
# reducer.py
import sys

previous_word = None
total = 0

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    if word != previous_word:
        if previous_word is not None:
            print(f"{previous_word}\t{total}")
        previous_word = word
        total = 0
    total += int(count)

if previous_word is not None:
    print(f"{previous_word}\t{total}")

Place the input in HDFS and run the streaming job:

printf 'hadoop hadoop pig\n' > words.txt
hdfs dfs -mkdir -p /user/interview/pig/input
hdfs dfs -put words.txt /user/interview/pig/input/words.txt
hdfs dfs -ls -R /user/interview/pig/input

hadoop jar $HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-*.jar \
  -files mapper.py,reducer.py \
  -mapper 'python3 mapper.py' \
  -reducer 'python3 reducer.py' \
  -input /user/interview/pig/input/words.txt \
  -output "/user/interview/pig/word-count-$$"

Python 3.12 must be installed and accessible on every node that executes the mapper or reducer.

Common Pitfalls

  • Global sorting: ORDER can be expensive at scale; partition data first when possible.

  • Skewed keys: A heavily repeated join or grouping key can overload one reducer.

  • Small files: Many tiny inputs or output part files increase metadata and scheduling overhead.

  • Existing output paths: Hadoop and Pig may reject an output location that already exists; use a unique path.

  • Incorrect data types: PigStorage does not perform application-level validation, so malformed records should be rejected before processing.

  • Unsafe aggregation: A combiner or pre-aggregation must preserve the final result.

35

How do you automate MySQL-to-HDFS imports with Sqoop?

Sqoop automates MySQL-to-HDFS ingestion by running JDBC-based bulk import jobs that write data into HDFS. Scheduling a parameterized script makes bulk and incremental loads repeatable, while introducing batch latency, source-database load, and operational complexity. Apache Sqoop is retired, so it is mainly appropriate for existing Hadoop installations.

Read full answer

Applies toApache Sqoop 1.4.7 (retired) on Hadoop 3.x; Python 3.12 for the streaming example

How It Works

  • HDFS: The NameNode stores metadata, while DataNodes store blocks. Hadoop 3.x defaults to a 128 MB block size and a replication factor of 3.

  • YARN: The ResourceManager allocates resources, NodeManagers run containers, and the ApplicationMaster coordinates the distributed job.

  • Hadoop 3.x: It adds erasure coding, and HDFS high availability can use multiple standby NameNodes.

  • MapReduce word count:

  • Map: Each mapper emits (word, 1) for every word.

  • Combiner: An optional combiner sums local counts before data transfer, reducing network traffic. It is an optimization, not a correctness requirement.

  • Partition and shuffle: A partitioner, by default a hash partitioner, assigns each key to a reducer. The framework sorts map output by key and groups matching words.

  • Reduce: Each reducer sums the grouped counts and writes the final totals.

  • Data locality: Hadoop schedules map tasks near the HDFS blocks holding their input, avoiding unnecessary remote reads.

  • Ecosystem: Hive provides SQL on Hadoop; HBase is a distributed wide-column store; Pig is legacy; Oozie schedules workflows; ZooKeeper coordinates distributed services; Spark is an in-memory engine that often runs on YARN and reads HDFS.

Automation Strategy

  1. Store the password in a protected file readable only by the Sqoop account, and use a read-only MySQL account.

  2. Give every run a unique output directory; Sqoop refuses to write into an existing directory unless replacement is explicitly managed.

  3. Invoke the script with cron, Airflow, Oozie, or another scheduler.

  4. Capture logs, monitor the YARN application, and make the scheduler fail when Sqoop returns a nonzero status.

  5. For recurring loads, use --incremental append --check-column order_id --last-value <watermark>. Choose a stable, monotonically increasing column and update the stored watermark only after success.

Modern Alternatives

Tool How it works Strengths Weaknesses Typical use
Sqoop Parallel JDBC scans into HDFS Simple and HDFS-native Retired and batch-oriented Existing Hadoop clusters
Spark Distributed JDBC reads and transformations Rich processing and faster pipelines More operational complexity Recurring ETL
Cloud services Managed ingestion into object storage or warehouses Less infrastructure maintenance Integration and cost constraints New cloud platforms

Many teams now use Spark, S3, GCS, ADLS, or cloud warehouses instead, but HDFS, YARN, data locality, and fault tolerance remain common interview topics.

Code Example

The password file must contain only the password and have mode 600. The split column is the table’s numeric primary key.

#!/usr/bin/env bash
set -euo pipefail

MYSQL_HOST="mysql.internal"
MYSQL_PORT="3306"
MYSQL_DATABASE="analytics"
MYSQL_TABLE="orders"
PASSWORD_FILE="/etc/sqoop/mysql-password.txt"
HDFS_BASE="/data/imports/orders"
RUN_ID="$(date -u +%Y%m%dT%H%M%S%NZ)-$$"
OUTPUT_DIR="${HDFS_BASE}/${RUN_ID}"

hdfs dfs -mkdir -p "${HDFS_BASE}"

sqoop import \
  --connect "jdbc:mysql://${MYSQL_HOST}:${MYSQL_PORT}/${MYSQL_DATABASE}" \
  --username "sqoop_import" \
  --password-file "${PASSWORD_FILE}" \
  --table "${MYSQL_TABLE}" \
  --columns "order_id,customer_id,order_total,created_at" \
  --split-by order_id \
  --fields-terminated-by '\t' \
  --null-string '\N' \
  --null-non-string '\N' \
  --batch \
  --m "${SQOOP_MAPPERS:-4}" \
  --output-dir "${OUTPUT_DIR}"

hdfs dfs -ls -R "${OUTPUT_DIR}"

For the required Hadoop Streaming example, save the following as mapper.py:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

Save reducer.py as follows. Hadoop Streaming provides sorted, grouped keys, so the reducer only retains one key’s count at a time.

#!/usr/bin/env python3
import sys

current_word = None
total = 0

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    if current_word is not None and word != current_word:
        print(f"{current_word}\t{total}")
        total = 0
    current_word = word
    total += int(count)

if current_word is not None:
    print(f"{current_word}\t{total}")

Create the input locally, upload it, and run both the local test and Hadoop Streaming job:

printf 'big data\nbig data\n' > input.txt

hdfs dfs -mkdir -p /data/inputs/wordcount
hdfs dfs -put -f input.txt /data/inputs/wordcount/
hdfs dfs -ls -R /data/inputs/wordcount/

# Expected output: big<TAB>2 and data<TAB>2
printf 'big data\nbig data\n' |
  python3.12 mapper.py |
  LC_ALL=C sort -k1,1 |
  python3.12 reducer.py

OUTPUT="/data/output/wordcount-$(date -u +%Y%m%dT%H%M%SZ)-$$"
hadoop jar "${HADOOP_HOME}"/share/hadoop/tools/lib/hadoop-streaming-*.jar \
  -files mapper.py,reducer.py \
  -input /data/inputs/wordcount \
  -output "${OUTPUT}" \
  -mapper 'python3.12 mapper.py' \
  -reducer 'python3.12 reducer.py'

Common Pitfalls

  • Overwriting output: Sqoop rejects an existing output directory; use a unique path or manage replacement explicitly.

  • Bad incremental logic: An incorrect watermark can duplicate or skip rows, especially with concurrent writes or updates.

  • Database overload: Too many mappers can exhaust MySQL connections; tune --m and monitor the database.

  • Secret exposure: Avoid --password, which can appear in command-line process listings.

  • Small files: Frequent incremental runs can create many tiny HDFS files; combine them before downstream processing.

  • Failure recovery: HDFS block replication and MapReduce task re-execution provide fault tolerance, but retries must be idempotent.

  • NameNode failure: Configure HDFS high availability with active and standby NameNodes in production.

Interview Tips

Be ready to explain the NameNode, DataNodes, ResourceManager, NodeManagers, and ApplicationMaster. Also explain the difference between replication and erasure coding, how combiners reduce shuffle traffic, and why incremental-import watermarks must be persisted only after a successful load.

36

How do I join two datasets with Spark?

A Spark join combines rows from two datasets using one or more equality keys, such as customer_id. DataFrame joins support several matching semantics, while Spark chooses an execution strategy that balances correctness, shuffle cost, memory usage, and scalability.

Read full answer

Applies toApache Spark 3.5.1–3.5.x and 4.x with PySpark (Python 3.12) on Hadoop 3.3+

How the Join Works

In a DataFrame join, Spark’s Catalyst optimizer analyzes the logical plan and selects a physical join strategy:

  1. Spark examines table statistics, estimated sizes, key distributions, and existing partitioning.

  2. For a shuffle-based join, Spark exchanges rows so keys with the same value are placed in the same partition. A sort-merge join also sorts each partition before merging it.

  3. Rows with equal keys are matched according to the selected join type.

  4. The resulting rows are returned as a new DataFrame.

For a large fact table and small dimension table, broadcast() replicates the small table across executors and avoids shuffling that side. The replicated table must fit comfortably in executor memory.

Join Types

Join type Rows returned
inner Rows with matches on both sides
left Every left row, with NULL for unmatched right-side columns
right Every right row, with NULL for unmatched left-side columns
outer All rows from both sides
semi Left columns for rows having a match
anti Left columns for rows without a match
cross Cartesian product; avoid unless the result is intentionally small

Use an inner join when only matching records are valid, and a left join when unmatched fact rows must be retained.

Performance Strategies

Strategy How it works Best use Main limitation
Broadcast hash join Replicates the smaller side Small lookup tables Replicated data consumes memory
Sort-merge join Exchanges, sorts, and merges partitions by key Large datasets on both sides Sorting, I/O, and shuffle overhead
Bucketed join Uses compatible bucket metadata to avoid a shuffle Repeated joins on large tables Requires identically bucketed catalog tables

Common controls include selecting useful columns before joining, broadcasting a genuinely small side, and persisting a repeatedly used dataset. Repartitioning can help only when its hash partitioning and partition count match downstream requirements; repartition itself performs a shuffle. Because Spark is lazy, cache() or persist() helps only when the DataFrame is reused and materialized.

Hadoop and Ecosystem Context

Spark commonly runs on YARN, whose ResourceManager allocates resources, NodeManagers manage containers, and an ApplicationMaster coordinates each application. It reads and writes HDFS, where the NameNode manages metadata and DataNodes store 128 MB blocks, normally with a replication factor of 3. Task failures are handled through lineage and re-execution; HDFS tolerates DataNode failures through block replicas or optional Hadoop 3 erasure coding.

Classic MapReduce word count maps each word to (word, 1), optionally runs a combiner within each mapper to pre-aggregate locally, uses a partitioner to route each key, shuffles and sorts by key, and then reducers sum the counts. Spark provides richer in-memory DAG execution while retaining similar distributed shuffle and data-locality principles.

In the Hadoop ecosystem, Hive provides SQL, HBase is a wide-column database, Pig is legacy, Sqoop is retired, Oozie manages workflows, and ZooKeeper provides coordination. Many organizations now use object storage, cloud warehouses, or Spark directly, but HDFS/YARN concepts remain relevant interview topics. Hadoop 3 also supports multiple standby NameNodes for high availability.

Code Example

import numpy as np
from pyspark.sql import SparkSession
from pyspark.sql.functions import broadcast

rng = np.random.default_rng(42)

spark = (
    SparkSession.builder
    .master("local[2]")
    .appName("spark-join-example")
    .config("spark.sql.shuffle.partitions", "4")
    .getOrCreate()
)

customer_ids = np.arange(1, 5)
customers = [
    (
        int(customer_id),
        f"Customer {customer_id}",
        int(rng.integers(10, 100)),
    )
    for customer_id in customer_ids
]

orders = [
    (
        order_id,
        int(rng.choice(customer_ids)),
        round(float(rng.uniform(20.0, 500.0)), 2),
    )
    for order_id in range(1, 9)
]

customers_df = spark.createDataFrame(
    customers,
    "customer_id INT, customer_name STRING, customer_score INT",
)

orders_df = spark.createDataFrame(
    orders,
    "order_id INT, customer_id INT, amount DOUBLE",
)

# Broadcast the small customer table to avoid shuffling that side.
joined_df = (
    orders_df.join(
        broadcast(customers_df),
        on="customer_id",
        how="inner",
    )
    .select(
        "order_id",
        "customer_id",
        "customer_name",
        "customer_score",
        "amount",
    )
    .orderBy("order_id")
)

joined_df.show(10, truncate=False)
spark.stop()

Common Pitfalls

  • Ambiguous columns: Qualify or rename duplicated columns before joining.

  • Accidental row multiplication: Verify that join keys are unique in the intended dimension table.

  • Data leakage: If joins construct features or labels, split the data before joining to prevent leakage.

  • Unnecessary shuffles: Broadcast a genuinely small side; repartition only when it reduces repeated work and matches the required partitioning.

  • Wrong join type: left, outer, and cross joins can increase result size substantially.

  • Memory pressure: Broadcasting oversized data can cause executor failures.

  • Small files: Repeated small outputs create metadata pressure on the NameNode and reduce HDFS efficiency.

Interviewers commonly ask why a broadcast join was chosen, how duplicate keys affect the result, how null keys behave, and how a Spark shuffle differs from a MapReduce shuffle.

37

How do you build an Oozie workflow for MapReduce and Hive?

An Oozie workflow is an XML-defined DAG that coordinates Hadoop actions, including MapReduce and Hive, through explicit dependencies. It solves batch-job scheduling on Hadoop/YARN, at the cost of tight coupling to Hadoop and less flexibility than modern cloud-native orchestrators.

Read full answer

Applies toApache Hadoop 3.3.x, Apache Oozie 5.2.x, Apache Hive 3.x, and Python 3.12

Workflow Design

  • Workflow and coordinator: An Oozie server executes a workflow DAG, while a coordinator schedules workflow instances according to a dataset definition.

  • HDFS: The NameNode manages metadata and DataNode processes store blocks. Defaults include 128 MB blocks and replication factor 3. Hadoop 3.x also supports erasure coding and HDFS high availability with multiple standby NameNode processes.

  • YARN: The ResourceManager allocates resources, NodeManager processes host containers, and an ApplicationMaster manages each application.

  • MapReduce: A mapper emits key-value pairs; an optional combiner aggregates map-side output; the partitioner assigns reducer buckets; shuffle transfers and sorts partitions by key; and the reducer merges final values. Map tasks are preferred near their HDFS input for data locality. Block replication and YARN task re-execution provide fault tolerance.

Component Role
Hive SQL layer over Hadoop data
HBase Distributed wide-column NoSQL store
Pig Legacy Hadoop batch scripting
Sqoop Retired bulk import/export connector
Oozie Hadoop workflow and coordinator scheduler
ZooKeeper Distributed coordination service
Spark Faster engine commonly using YARN and HDFS

Code Example

Save the workflow as word-count-workflow.xml. The -files option localizes both Python scripts into each task’s working directory.

<?xml version="1.0" encoding="UTF-8"?>
<workflow-app name="word-count"
             xmlns="http://oozie.apache.org/ns/workflow/1.0">
  <start>
    <action name="count-words">
      <mapreduce>
        <name-node>hdfs://namenode:8020</name-node>
        <streaming>
          <mapper>python3.12 word_count_mapper.py</mapper>
          <reducer>python3.12 word_count_reducer.py</reducer>
          <combiner>python3.12 word_count_reducer.py</combiner>
        </streaming>
        <arg>-files</arg>
        <arg>hdfs://namenode:8020/user/oozie/workflows/wordcount/scripts/word_count_mapper.py,hdfs://namenode:8020/user/oozie/workflows/wordcount/scripts/word_count_reducer.py</arg>
        <arg>-input</arg>
        <arg>/user/oozie/workflows/wordcount/input</arg>
        <arg>-output</arg>
        <arg>/user/oozie/workflows/wordcount/counts/${wfid}</arg>
      </mapreduce>
      <transition type="next" to="aggregate-with-hive"/>
    </action>
  </start>

  <action name="aggregate-with-hive">
    <hive>
      <configuration>
        <property>
          <name>word_count_location</name>
          <value>/user/oozie/workflows/wordcount/counts/${wfid}</value>
        </property>
      </configuration>
      <script>/user/oozie/workflows/wordcount/scripts/aggregate.hql</script>
    </hive>
    <transition type="next" to="done"/>
  </action>

  <end name="done"/>
</workflow-app>

word_count_mapper.py:

import sys

for line in sys.stdin:
    for word in line.lower().split():
        sys.stdout.write(f"{word}\t1\n")

word_count_reducer.py also works as the map-side combiner:

import sys
from collections import Counter

counts = Counter()
for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    counts[word] += int(value)

for word in sorted(counts):
    print(f"{word}\t{counts[word]}")

aggregate.hql:

DROP TABLE IF EXISTS word_counts_stage;

CREATE EXTERNAL TABLE word_counts_stage (
    word STRING,
    cnt BIGINT
)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
STORED AS TEXTFILE
LOCATION '${hiveconf:word_count_location}';

CREATE TABLE IF NOT EXISTS word_counts (
    word STRING,
    total BIGINT
) STORED AS TEXTFILE;

INSERT OVERWRITE TABLE word_counts
SELECT word, SUM(cnt)
FROM word_counts_stage
WHERE word IS NOT NULL AND word <> ''
GROUP BY word;

Deployment and Scheduling

Upload the scripts, input, and workflow definitions:

hdfs dfs -mkdir -p /user/oozie/workflows/wordcount/scripts \
  /user/oozie/workflows/wordcount/input

printf 'hadoop mapreduce\nhadoop streaming\n' > input.txt

hdfs dfs -put word_count_mapper.py /user/oozie/workflows/wordcount/scripts/
hdfs dfs -put word_count_reducer.py /user/oozie/workflows/wordcount/scripts/
hdfs dfs -put aggregate.hql /user/oozie/workflows/wordcount/scripts/
hdfs dfs -put input.txt /user/oozie/workflows/wordcount/input/
hdfs dfs -put word-count-workflow.xml /user/oozie/workflows/wordcount/
hdfs dfs -put word-count-coordinator.xml /user/oozie/workflows/wordcount/
hdfs dfs -ls -R /user/oozie/workflows/wordcount

Submit a one-off workflow with oozie -oozie http://oozie-server:11000/ -config word-count-workflow.xml -run.

For daily execution, replace these example UTC dates with the desired window and save the definition as word-count-coordinator.xml:

<coordinator-app name="daily-word-count"
                 xmlns="http://oozie.apache.org/ns/coordinator/1.0">
  <start-time>2026-01-01T00:00:00Z</start-time>
  <end-time>2026-12-31T23:59:59Z</end-time>
  <datasets>
    <dataset name="daily" frequency="1 day"
             start="2026-01-01T00:00:00Z"
             end="2026-12-31T23:59:59Z"/>
  </datasets>
  <actions>
    <action name="launch-workflow">
      <workflow app-path="/user/oozie/workflows/wordcount/word-count-workflow.xml"/>
    </action>
  </actions>
</coordinator-app>

Submit it with oozie -oozie http://oozie-server:11000/ -config word-count-coordinator.xml -submit.

Common Pitfalls

  • Existing output: MapReduce rejects an existing output directory. ${wfid} gives each workflow instance a unique path.

  • Missing dependencies: Every YARN container needs python3.12; the mapper and reducer must also be readable through HDFS.

  • Incorrect dependencies: Oozie does not infer data lineage, so transitions must encode the MapReduce-before-Hive dependency.

  • Small files and Hive locations: Many tiny HDFS files burden the NameNode, while external-table locations, schemas, metastore permissions, and MapReduce output must agree.

  • Current role: Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses instead of Oozie, but HDFS/YARN concepts such as data locality, replication, fault tolerance, and DAG scheduling remain important.

38

How do custom Hadoop InputFormat and OutputFormat classes work?

Custom Hadoop InputFormat and OutputFormat classes plug format-specific behavior into MapReduce: an input format chooses splits and creates record readers, while an output format creates record writers and controls result serialization. They solve custom parsing, splitting, compression, and layout requirements at the cost of extra maintenance and reduced compatibility with standard tools.

Read full answer

Applies toApache Hadoop 3.4.x (Java 11; Hadoop Streaming examples use Python 3.12)

How It Works

  • Input lifecycle: InputFormat.getSplits() runs on the client and returns logical InputSplit objects. Each task creates a RecordReader, calls initialize(), repeatedly calls nextKeyValue(), and finally calls close().

  • Split and record policy: A custom input format can override getSplits() or, as below, isSplitable() to control split size and boundaries. A completely custom format normally supplies a RecordReader that converts split bytes into records.

  • Output lifecycle: OutputFormat.checkOutputSpecs() validates output settings client-side. In each task, getRecordWriter() creates a writer that receives key-value pairs; write() serializes each pair and close() finishes the attempt. The output committer publishes successful results.

  • Platform context: HDFS uses a NameNode for metadata and DataNodes for blocks; defaults include 128 MB blocks and replication factor 3. Hadoop 3 adds erasure coding and multiple standby NameNodes. YARN’s ResourceManager, NodeManagers, and ApplicationMaster run MapReduce, with task retries and HDFS replication providing fault tolerance.

  • Ecosystem and MapReduce: Hive provides SQL, HBase is a wide-column store, Pig is legacy, Sqoop is retired, Oozie handles workflows, ZooKeeper provides coordination, and Spark is an in-memory engine often running on YARN and reading HDFS. In word count, the mapper emits (word, 1), a partitioner routes each key, shuffle and sort group keys, an optional combiner preaggregates locally, reducers sum values, and data locality schedules tasks near their splits. Cloud object stores and warehouses are common today, but locality and fault tolerance remain important concepts.

Code Example

This custom input format makes each file indivisible, while the output format prefixes each value and preserves Hadoop’s output commit protocol.

import java.io.DataOutputStream;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.JobContext;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.RecordWriter;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.input.TextInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat;

public class CustomTextIO {
    public static class WholeFileInputFormat extends TextInputFormat {
        @Override
        protected boolean isSplitable(JobContext context, Path file) {
            return false;
        }
    }

    public static class PrefixedTextOutputFormat
            extends TextOutputFormat<Text, Text> {
        @Override
        public RecordWriter<Text, Text> getRecordWriter(
                TaskAttemptContext context, DataOutputStream out) {
            return new RecordWriter<Text, Text>() {
                @Override
                public void write(Text key, Text value) throws IOException {
                    out.write(key.toString().getBytes(StandardCharsets.UTF_8));
                    out.write('\t');
                    out.write(("PREFIX:" + value)
                            .getBytes(StandardCharsets.UTF_8));
                    out.write('\n');
                }

                @Override
                public void close() throws IOException {
                    out.close();
                }
            };
        }
    }

    public static class PassThroughMapper
            extends Mapper<Text, Text, Text, Text> {
        @Override
        protected void map(Text key, Text value, Context context)
                throws IOException, InterruptedException {
            context.write(key, value);
        }
    }

    public static void main(String[] args) throws Exception {
        if (args.length != 2) {
            throw new IllegalArgumentException(
                    "Usage: CustomTextIO <input> <output>");
        }
        Configuration configuration = new Configuration();
        Job job = Job.getInstance(configuration, "custom-text-io");
        job.setInputFormatClass(WholeFileInputFormat.class);
        job.setOutputFormatClass(PrefixedTextOutputFormat.class);
        job.setMapperClass(PassThroughMapper.class);
        job.setNumReduceTasks(0);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(Text.class);
        FileInputFormat.addInputPath(job, new Path(args[0]));
        FileOutputFormat.setOutputPath(job, new Path(args[1]));
        System.exit(job.waitCompletion(true) ? 0 : 1);
    }
}
export HADOOP_CLASSPATH="$(hadoop classpath)"
javac -cp "$HADOOP_CLASSPATH" CustomTextIO.java
jar cf custom-text-io.jar CustomTextIO*.class

printf 'alpha beta\nbeta gamma\n' \
  | hdfs dfs -put -f - "/user/$USER/custom-input.txt"
hdfs dfs -ls -h "/user/$USER/custom-input.txt"

hadoop jar custom-text-io.jar CustomTextIO \
  "/user/$USER/custom-input.txt" "/user/$USER/custom-output"

Hadoop Streaming Example

Save these as mapper.py and reducer.py. Streaming sorts mapper output by key before invoking the reducer.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.strip().split():
        print(f"{word}\t1")
# reducer.py
import sys

current_word = None
count = 0
for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    count += int(value)
    if current_word is not None and word != current_word:
        print(f"{current_word}\t{count}")
        count = 0
    current_word = word

if current_word is not None:
    print(f"{current_word}\t{count}")
hadoop jar "$HADOOP_HOME/share/hadoop/tools/lib/hadoop-streaming-"*.jar \
  -input "/user/$USER/custom-input.txt" \
  -output "/user/$USER/streaming-output" \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py"

Common Pitfalls

  • Wrong key interpretation: TextInputFormat keys are byte offsets, not line numbers.

  • Skewed splits: One split per file can overload one task when file sizes differ substantially.

  • Invalid output paths: Final output directories generally must not already exist.

  • Small files: Many tiny files increase NameNode metadata and scheduling overhead.

  • Unsafe output writing: Write through RecordWriter and preserve the output committer rather than publishing directly to final HDFS paths.

39

How does HDFS erasure coding differ from replication?

Erasure coding in HDFS stores a logical block as data blocks plus parity blocks, allowing missing blocks to be reconstructed from surviving data and parity. It reduces storage overhead compared with replication, but recovery requires additional computation and network I/O and introduces greater operational complexity.

Read full answer

Applies toApache Hadoop 3.x

How Erasure Coding Works

HDFS groups blocks into stripe groups. Each stripe contains original data blocks and parity blocks generated by an erasure-coding scheme. The blocks are distributed across DataNodes so that losing one DataNode does not necessarily make the stripe unrecoverable.

A built-in Hadoop 3.x policy is RS-6-3-3-1024k:

  • 6 data blocks: contain the original data.

  • 3 parity blocks: contain recovery information.

  • 3 recovery units: the third 3 in the scheme name.

  • 9 stored blocks total: six data blocks plus three parity blocks.

The RS-6-3-3 scheme can reconstruct up to three missing blocks per stripe, provided enough surviving blocks remain and the failure domains do not exceed the configured placement policy. Its storage overhead is 3 / 6 = 50%. Parity blocks are not complete copies and cannot individually replace an arbitrary missing data block without reconstruction.

Erasure Coding vs. Replication

Aspect Erasure coding Replication
Storage model Data blocks plus parity blocks Identical copies of each block
Typical HDFS configuration A policy such as RS-6-3-3-1024k Replication factor of 3
Storage overhead Often lower than replication 3 / 1 = 200% for three copies
Fault tolerance Reconstructs missing blocks using data and parity Recovers a block by copying another replica
Recovery cost More CPU and network I/O because decoding is required Usually cheaper because recovery is a direct copy
Write cost Requires encoding and writing parity Writes multiple copies
Operational complexity More complex because policies and stripe recovery must be managed Simpler and widely supported
Best fit Large, cold, or infrequently accessed data Frequently accessed or latency-sensitive data

Replication stores three copies of each default 128 MB HDFS block, normally on separate DataNodes. Erasure coding trades more expensive recovery for lower storage consumption.

HDFS supports mixed policies: administrators can configure erasure coding for selected directories or paths while retaining replication elsewhere. This is not automatic hot/cold tiering, because HDFS does not move files between policies solely because their access frequency changes.

Fault Tolerance and Block Placement

Both mechanisms tolerate DataNode and block failures, but their recovery paths differ. HDFS detects missing or corrupt stored blocks and schedules recovery. For replication, a surviving replica is copied to restore the required replica count. For erasure coding, HDFS reads enough surviving data and parity blocks, reconstructs the missing coded blocks, and writes replacements.

EC recovery can therefore generate substantial temporary network traffic and CPU usage, especially when several blocks fail together. The recovery guarantee depends on the selected policy, block placement, and whether failures are spread across nodes and racks.

Advantages and Limitations

Advantages

  • Lower storage overhead: Usually the main reason for adopting erasure coding.

  • Configurable fault tolerance: Different policies provide different numbers of recoverable block losses.

  • Directory-level policy selection: Different paths can use replication or erasure coding.

  • Useful for long-term storage: It is economical for large datasets accessed infrequently.

Limitations

  • Higher recovery cost: Missing blocks require decoding and reconstruction.

  • More administration: Policies, placement, rack awareness, and recovery load must be managed.

  • Potential performance impact: Encoding and recovery can compete with normal workloads.

  • Not a complete replacement for replication: Files outside directories configured for erasure coding continue to use replication.

Common Pitfalls

  • Do not call parity a backup: Parity reconstructs losses within a stripe but does not replace independent backups or disaster recovery.

  • Do not assume durability is automatically identical: Compare the policies, rack-aware placement, and correlated-failure tolerance.

  • Do not ignore recovery load: Simultaneous failures can cause large reconstruction traffic.

  • Do not confuse block recovery with NameNode high availability: Erasure coding protects stored data blocks, while Hadoop 3.x NameNode high availability, including multiple standby NameNodes, protects metadata-service availability.

  • Interview follow-up: Common questions ask why organizations choose erasure coding for cold data, how RS-6-3-3 tolerates three block losses, and how reconstruction differs from replication recovery.

40

How does Hadoop use data locality for faster processing?

Data locality is Hadoop’s practice of scheduling computation near the HDFS data it needs, reducing network transfers and remote disk access. It improves MapReduce performance by running map tasks on nodes that already contain the input blocks. The main trade-off is tighter task placement: locality can conflict with workload balancing, so YARN may fall back to rack-local or off-rack execution when necessary.

Read full answer

Applies toApache Hadoop 3.x (HDFS, YARN, and MapReduce)

How Data Locality Works

  • HDFS stores large files as blocks: The default block size is 128 MB. In the default replicated configuration, each block normally has three replicas across distinct DataNodes.

  • The NameNode tracks metadata: It records which DataNodes contain each block and its replicas.

  • YARN makes location-aware scheduling decisions: The ResourceManager’s scheduler uses NodeManager reports and HDFS block-location information. The ApplicationMaster requests containers with locality preferences, and NodeManagers launch them.

  • Replication provides alternatives: For replicated blocks, if the ideal node is busy or fails, Hadoop can schedule the task near another replica.

Tasks commonly receive blocks at different locality levels:

Locality level Behavior Benefit
Node-local Runs on a node containing the input block Lowest network and remote-disk cost
Rack-local Runs on another node in the same rack Uses the rack’s high-bandwidth network
Off-rack Runs on a node in another rack Lowest locality and highest network cost

MapReduce and Data Locality

For a word-count job:

  1. Map: HDFS input splits are assigned to map tasks. The scheduler places each task on a node containing that split when possible.

  2. Combine: An optional map-side aggregation reduces downstream traffic when the operation preserves the required semantics.

  3. Partition, sort, and shuffle: A partitioner selects the reducer for each key. Map outputs are sorted by key and transferred to their reducers.

  4. Reduce: Each reducer counts the values associated with its keys and writes the result to HDFS.

Map tasks benefit most from locality because they read the largest volume of source data. Reducers consume map output from many map tasks, so they usually have weaker data locality and rely more heavily on network transfer.

Performance and Fault Tolerance

  • Less network traffic: Reading blocks locally avoids transferring large volumes of input across the cluster.

  • Lower remote-I/O latency: Local HDFS reads generally avoid remote disk and network access.

  • Task re-execution: If a task fails, Hadoop reruns it, preferably near another replica of its input.

  • Scheduling flexibility: YARN falls back to less-local locations when local resources are insufficient.

  • Potential imbalance: Strict locality preference can leave some nodes underused, so YARN balances locality against cluster utilization.

  • Default replication: The usual three replicas provide both fault tolerance and alternative scheduling locations.

  • Erasure coding: Hadoop 3.x can store data in stripes across DataNodes, reducing storage overhead compared with three-way replication, although reconstruction can require additional I/O. Locality must account for the nodes holding the stripe’s chunks.

  • NameNode high availability: Hadoop 3.x supports an active NameNode with multiple standby NameNodes, allowing metadata service to survive a NameNode failure.

Modern Relevance

HDFS and YARN remain important interview concepts because they explain distributed storage, data locality, fault tolerance, and cluster scheduling. However, many organizations now use cloud object stores such as S3, GCS, or ADLS, along with Spark or managed warehouses, instead of Hadoop clusters. Spark can prefer executors near relevant HDFS blocks, but object storage has no DataNode-local placement, so executor-local caching and reducing repeated remote reads become more important.

Common Pitfalls

  • Assuming every map task is node-local: Data placement, resource availability, and replication can force rack-local or off-rack execution.

  • Confusing a block with a MapReduce split: An input split is a scheduling unit and may not correspond exactly to an HDFS block.

  • Ignoring shuffle traffic: Map tasks are locality-friendly, but reducer inputs still cross the network.

  • Overlooking the small-files problem: Many tiny files increase NameNode metadata, scheduling, and connection overhead; compaction or larger files usually improve performance.

  • Forgetting fault tolerance: Block replication or erasure coding, together with task re-execution, allows processing to continue after storage or node failures.

  • Useful follow-up questions: How does YARN choose among replicas, how are rack-aware policies configured, how does shuffling affect locality, and how does Spark differ when reading HDFS versus object storage?

41

How does Hadoop read and write different file formats?

File-format support in Hadoop is provided by pluggable InputFormat/RecordReader and OutputFormat/RecordWriter components. MapReduce jobs work with key-value records, while Avro, Parquet, and ORC provide schema-aware implementations. The trade-off is flexible row-oriented formats versus faster columnar scans, with columnar files adding metadata and write overhead.

Read full answer

Applies toApache Hadoop 3.x (Python 3.12 Streaming example)

How Format Support Works

  • InputFormat: Selects input files and divides file-based inputs into splits.

  • RecordReader: Converts one split into key-value records consumed by a mapper.

  • OutputFormat: Defines the output representation and creates a RecordWriter; an OutputCommitter publishes the completed output.

  • Partitioner: Separately routes each key to a reducer, allowing different input and output formats.

  • Format libraries: Avro, Parquet, and ORC supply schema-aware readers and writers used directly or through Hive and Spark.

  • Compression: This is separate from record format. For example, Parquet can contain compressed column data while remaining a Parquet file.

A job can read Avro, SequenceFile, or Parquet and write another format when the mapper, reducer, readers, and writers use compatible key-value types.

Common Hadoop File Formats

Format Main characteristics Trade-offs and typical use
Text Human-readable and line-oriented No embedded schema; useful for logs and simple interchange
SequenceFile Hadoop binary key-value format with record boundaries Efficient for MapReduce; less useful for column-oriented analytics
Avro Row-oriented, schema-based serialization with schema resolution Strong for pipelines and evolving schemas; less efficient for selective column reads
Parquet Typed columnar storage with compression, statistics, and predicate pushdown Excellent for analytical scans; affected by small files and frequent updates
ORC Columnar format with nested types and statistics Strong for large Hive-style analytics
Compressed text Text compressed with a supported codec Saves storage; splitability depends on the codec

Hadoop and MapReduce Context

HDFS stores files in blocks of 128 MB by default with a replication factor of 3 by default. The NameNode stores metadata, while DataNodes store blocks. Replication and task re-execution provide fault tolerance; Hadoop 3.x also supports erasure coding and multiple standby NameNodes for high availability.

In YARN, the ResourceManager allocates resources, NodeManagers manage worker resources, and the ApplicationMaster coordinates each application.

A word-count MapReduce job follows these steps:

  1. Map: Read text locally and emit each word with a count of 1.

  2. Combiner: Optionally aggregate map-side counts to reduce network traffic.

  3. Shuffle and sort: Framework workers transfer, sort, and merge records; the Partitioner sends equal keys to the same reducer.

  4. Reduce: Sum each key’s values and write the results.

Data locality places computation near the input blocks. The ecosystem includes Hive for SQL, HBase for wide-column storage, legacy Pig, retired Sqoop, Oozie for workflows, ZooKeeper for coordination, and Spark as an in-memory engine that often runs on YARN and reads HDFS.

Code Example

The following Hadoop Streaming mapper and reducer implement word count. YARN containers must provide Python 3.12 as python3, and HADOOP_STREAMING_JAR must identify the local streaming JAR.

# mapper.py
import sys

for line in sys.stdin:
    for token in line.lower().split():
        word = "".join(character for character in token if character.isalnum())
        if word:
            print(f"{word}\t1")
# reducer.py
import sys

current_word = None
count = 0


def emit(word, total):
    if word is not None:
        print(f"{word}\t{total}")


for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    value = int(value)

    if current_word is not None and word != current_word:
        emit(current_word, count)
        count = 0

    current_word = word
    count += value

emit(current_word, count)
printf 'hadoop hadoop spark\nspark spark\n' > input.txt

hdfs dfs -mkdir -p /tmp/wordcount/input
hdfs dfs -put -f input.txt /tmp/wordcount/input/
hdfs dfs -ls -h /tmp/wordcount/input

hdfs dfs -rm -r -f /tmp/wordcount/output
hadoop jar "$HADOOP_STREAMING_JAR" \
  -input /tmp/wordcount/input \
  -output /tmp/wordcount/output \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -numReduce 1 \
  -files mapper.py,reducer.py

hdfs dfs -cat /tmp/wordcount/output/part-*
# Deterministic result:
# hadoop 2
# spark 3

Best Practices

  • Prefer Parquet or ORC for repeated scans, aggregations, and data-warehouse workloads.

  • Use Avro or SequenceFile when complete records or Hadoop-native key-value semantics are required.

  • Define schema-evolution policies explicitly because compatibility rules differ by format.

  • Compact small files, which increase metadata and scheduling overhead.

  • Many teams now use S3, GCS, or ADLS with Spark or cloud warehouses, but HDFS, YARN, data locality, replication, and fault-tolerance concepts remain important interview topics.

Common Pitfalls

  • A standalone gzip file is generally not efficiently splittable; bzip2 is splittable, while Snappy is normally used inside container formats.

  • The file extension does not select the reader; the configured InputFormat and its libraries do.

  • Ignoring schema changes or inferring schemas from samples can corrupt downstream processing.

  • Too many reducers or one reducer can respectively create overhead or limit parallelism.

Interview follow-ups commonly concern predicate pushdown, schema evolution, small-file compaction, replication versus erasure coding, NameNode high availability, and when object storage is preferable.

42

What is Hadoop Federation and how does it scale clusters?

Hadoop Federation allows HDFS to operate multiple independent NameNode namespaces, distributing metadata management across several NameNodes instead of concentrating it in one server. It addresses metadata scalability and namespace-isolation bottlenecks, with the main trade-off being greater configuration and operational complexity. Storage and compute still scale separately by adding DataNodes and YARN NodeManagers.

Read full answer

Applies toApache Hadoop 3.x

Federation vs. High Availability

  • HDFS Federation: Multiple NameNodes manage different namespaces. A namespace normally corresponds to a logical group of directories, and clients use the appropriate NameNode for its paths.

  • Client routing: Clients address namespaces through their configured nameservices. ViewFS can also mount multiple namespaces into a single logical directory hierarchy.

  • DataNode independence: A DataNode can store blocks belonging to multiple namespaces. The NameNodes maintain metadata, while clients transfer actual data directly between clients and DataNodes.

  • High availability: Federation does not itself make a NameNode redundant. HA is configured separately within a namespace using an active NameNode, standby NameNodes, shared JournalNodes, and typically ZooKeeper-based automatic failover.

  • Multiple standbys: Hadoop 3.x supports multiple standby NameNodes within a nameservice, allowing another standby to be promoted if the active node fails.

  • Key distinction: Federation scales metadata horizontally across namespaces; HA provides service availability within one namespace. They can be used together.

How Cluster Scaling Works

Layer Scaling approach Main constraint
HDFS storage Add DataNodes and disk capacity Network throughput and disk failures
HDFS metadata Add namespaces and their NameNodes Namespace configuration and client routing
YARN compute Add NodeManagers and cluster resources ResourceManager scheduling and shuffle load
MapReduce processing Add input splits and parallel tasks Data skew and reducer bottlenecks

Federation does not automatically split an existing namespace. Administrators deliberately divide directories among namespaces so that each NameNode manages a manageable amount of metadata.

Hadoop 3.x commonly uses a 128 MB HDFS block size and a replication factor of 3. Replication provides fault tolerance through copies on different DataNodes. Erasure coding can reduce storage overhead, but it requires additional network and CPU work to reconstruct missing data.

YARN and MapReduce Execution

  • ResourceManager: Allocates cluster resources globally.

  • NodeManager: Manages resources and containers on one worker node.

  • ApplicationMaster: Coordinates one application and requests containers.

  • MapReduce: Runs map and reduce tasks as YARN applications.

For a word-count job, the execution sequence is:

  1. HDFS input is divided into splits, and mapper tasks process the splits close to the stored data when possible.

  2. Each mapper emits a word and a count of 1.

  3. A combiner may aggregate counts locally on the mapper side before intermediate results enter the network shuffle.

  4. A partitioner selects the reducer for each key; the default partitioner uses a hash of the key.

  5. The framework transfers and sorts intermediate records by key.

  6. Reducers sum the values for each word.

This design uses data locality: map tasks are scheduled on nodes holding the relevant input blocks. Reducers, however, commonly receive shuffled intermediate data from multiple map tasks.

Ecosystem and Operational Considerations

  • Hive provides SQL access to data stored in Hadoop.

  • HBase is a distributed wide-column NoSQL database.

  • Pig is a legacy data-flow platform.

  • Sqoop is a retired Apache project.

  • Oozie coordinates Hadoop workflows.

  • ZooKeeper provides distributed coordination.

  • Spark is a faster, primarily in-memory processing engine that commonly reads HDFS and runs on YARN.

Many organizations now use cloud object stores such as S3, GCS, or ADLS, along with Spark or cloud warehouses. Nevertheless, HDFS, YARN, data locality, replication, and task re-execution remain important distributed-systems concepts.

Common Pitfalls

  • Confusing federation with replication: Federation distributes namespace metadata; replication stores multiple copies of data blocks.

  • Ignoring storage bottlenecks: More NameNodes do not add HDFS capacity unless DataNodes and disks are also added.

  • Allowing small files: Many tiny files increase NameNode metadata and scheduling overhead. Compact them or combine them into larger files.

  • Overloading the ResourceManager: Adding compute nodes can expose scheduler, network, and shuffle bottlenecks.

  • Ignoring data skew: Uneven key distribution can leave some reducers idle while others process excessive data.

  • Assuming data loss is impossible: Replication and task re-execution improve fault tolerance, but capacity planning, monitoring, and recovery procedures are still required.

43

What does a JournalNode do in HDFS high availability?

A JournalNode is a dedicated Hadoop daemon that stores shared HDFS edit-log records for the Active and Standby NameNodes. This allows the Standby NameNode to remain synchronized and enables a new Active NameNode to recover safely during failover. The main trade-off is stronger metadata consistency and automated recovery in exchange for additional processes, operational complexity, and dependence on a JournalNode quorum.

Read full answer

Applies toApache Hadoop 3.x

How It Works

HDFS high availability uses the Quorum Journal Manager (QJM) to share namespace edit logs between NameNodes:

  • The Active NameNode handles client requests and writes namespace transactions, such as file creation, deletion, rename, and permission changes, to its local edit log.

  • It also sends those transactions to a quorum of JournalNodes, which persist the shared edit records on their local disks.

  • The Standby NameNode continuously follows the shared edits so it can take over without manually copying metadata from the Active NameNode.

  • During failover, the promoted NameNode retrieves and validates the latest committed edits from the JournalNodes before serving client requests.

A typical deployment uses three JournalNodes on independent machines. A majority quorum allows the system to tolerate one JournalNode failure while preventing two NameNodes from independently committing conflicting namespace changes.

Component Responsibilities

Component Responsibility
Active NameNode Handles client metadata operations and coordinates namespace transactions
Standby NameNode Tracks the Active NameNode and becomes ready for promotion
JournalNode Persists shared edit-log records and participates in the quorum
DataNode Stores HDFS data blocks, normally with a replication factor of 3
ZooKeeper Coordinates automatic failover and detects the active node; it does not store HDFS metadata

JournalNodes store metadata edit logs, not user file contents or data blocks. Their storage requirements are therefore separate from the HDFS block-storage layer. The standard HDFS data-block size of 128 MB does not determine the size or behavior of JournalNode edit segments.

Benefits

  • Consistent metadata state: The Standby NameNode receives the same committed edit stream as the Active NameNode rather than relying on an ad hoc file-copying process.

  • Fault-tolerant edit storage: With three JournalNodes, the quorum continues operating if one node fails.

  • Reliable failover: A promoted NameNode can retrieve shared edits needed to complete recovery and remain consistent with the last committed namespace state.

  • Simpler synchronization: The shared journal replaces direct Active-to-Standby edit-log copying and associated operational scripts.

  • Independent failure domains: JournalNodes can run on separate machines so that a single host failure does not remove the entire journal service.

Hadoop 3.x also supports multiple Standby NameNodes, while the JournalNode quorum continues to provide the shared edit history required for failover.

Limitations and Requirements

  • Quorum availability matters: If too many JournalNodes are unavailable, the Active NameNode cannot safely commit normal namespace changes, and failover recovery may be blocked.

  • Additional infrastructure: Each JournalNode requires its own process, local storage, monitoring, and preferably an independent machine or failure domain.

  • Synchronization overhead: Every relevant namespace transaction must be sent to the journal quorum, adding I/O and some latency.

  • Not full namespace storage by itself: JournalNodes primarily store edit segments; a NameNode also relies on filesystem images and other metadata to reconstruct its complete namespace.

  • Split-brain protection still matters: JournalNode coordination does not replace the fencing and automatic-failover mechanisms provided by ZooKeeper and the HDFS failover process.

Interview Tips

A strong answer should distinguish JournalNodes from DataNodes and Standby NameNodes. Emphasize that JournalNodes provide a quorum-backed shared edit log, not HDFS block storage. Common follow-up questions include:

  • Why are an odd number of JournalNodes normally deployed?

  • What happens when one of three JournalNodes fails?

  • What happens if the quorum is lost?

  • How does the promoted NameNode obtain the latest metadata?

  • How does JournalNode storage differ from DataNode block replication?

  • What additional role does ZooKeeper play during automatic failover?

The key phrase is: JournalNodes maintain a quorum-backed shared namespace edit log so HDFS can fail over without manually synchronizing the Active and Standby NameNodes.

44

How do small files degrade HDFS and how can teams mitigate it?

The small-file problem occurs when HDFS contains many files much smaller than its default 128 MB block size. These files consume disproportionate NameNode metadata, MapReduce scheduling resources, and storage I/O, even when cluster capacity appears available. Compaction improves efficiency, but it adds processing cost, can delay data availability, and must balance larger files against the parallelism needed by downstream jobs.

Read full answer

Applies toApache Hadoop 3.x with Python 3.12

Performance Impact

  • NameNode overhead: Every file requires namespace metadata and normally block metadata. Millions of small files increase memory pressure, fsimage and edit-log size, metadata RPC traffic, and NameNode startup or recovery time. NameNode HA and multiple standby NameNodes improve availability but do not eliminate this cost.

  • MapReduce and YARN overhead: Each input file normally produces one or more map splits. Numerous small files can create excessive YARN containers, scheduler requests, JVM startup costs, and task re-execution overhead.

  • Disk and network inefficiency: Opening, reading, and closing each file has fixed costs. Replication under the default replication factor of three adds write traffic and metadata without creating additional parallel processing capacity.

  • Ecosystem impact: Hive and Spark jobs, commonly running on YARN and reading HDFS, may spend significant time listing inputs and launching tasks rather than processing data.

  • Fault tolerance remains costly: Block replication and task re-execution preserve reliability, but neither removes the small-file overhead.

Mitigation Strategies

Approach How it works Advantages Trade-offs
Buffer at ingestion Accumulate records and write periodically instead of creating one file per event Prevents the problem at its source Introduces latency and risks data loss without proper buffering
Merge or compact Read many inputs and write fewer, target-sized part files Removes many file and block entries Requires temporary space, compute, and careful validation
Use archive files Pack files into an archive supported by Hadoop tools Reduces visible file count Reader compatibility and incremental updates may be inconvenient
Write larger partition files Store one SequenceFile, Parquet, or ORC file per logical partition Improves sequential reads and reduces scanner overhead File count must still be controlled; the format alone does not prevent small files

A useful compaction target is commonly one or a few HDFS blocks, but it should reflect access patterns. Extremely large files reduce task parallelism, while very small files retain the original problem.

Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses instead of HDFS/YARN, so new workloads should compare object-listing behavior and cost rather than assume HDFS. HDFS/YARN concepts such as distributed storage, data locality, and fault tolerance remain common interview topics and also apply to managed Hadoop services.

Operational Approach

  1. Measure file count, average size, NameNode load, and map-task count using tools such as hdfs dfs -count, hdfs fsck, and hdfs dfsadmin -report.

  2. Select target output sizes based on downstream parallelism, access frequency, and partition-level compaction requirements.

  3. Write compacted files to a temporary sibling location, validate row counts or checksums, and publish them using an HDFS rename.

  4. Monitor the rate at which new small files are created; ingestion controls are more sustainable than continuous reactive compaction.

MapReduce Walkthrough

  1. Split and map: HDFS input is divided into splits, and normally every small file produces at least one map task. Map tasks are scheduled near their blocks when data locality permits.

  2. Emit intermediate pairs: The mapper writes each word and a count of 1 as a tab-separated pair.

  3. Partition and combine: A partitioner routes each key to a reducer. An optional combiner aggregates local word counts before the shuffle, reducing transferred data when aggregation is associative.

  4. Shuffle and sort: The shuffle stage moves map output to reducers and groups it by key.

  5. Reduce: Each reducer sums the values for a key. One reducer creates one output part file; multiple reducers create multiple part files, which must be balanced against parallelism and output-file size.

Code Example

The following Hadoop Streaming mapper and reducer implement word count. Using one reducer produces one output part file, illustrating how reducer count affects output-file cardinality.

mapper.py:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

reducer.py:

#!/usr/bin/env python3
import sys

current_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t")
    value = int(value)

    if current_word is not None and word != current_word:
        print(f"{current_word}\t{count}")
        count = 0

    current_word = word
    count += value

if current_word is not None:
    print(f"{current_word}\t{count}")

Store both scripts locally and run them with Hadoop Streaming. The worker nodes must have Python 3.12 installed and available as python3.

printf 'hadoop hdfs\nhadoop yarn\n' > input.txt

hdfs dfs -mkdir -p input
hdfs dfs -put input.txt input/input.txt
hdfs dfs -ls -h input

hadoop jar "$HADOOP_HOME"/share/hadoop/tools/lib/hadoop-streaming-*.jar \
  -D mapreduce.job.reduces=1 \
  -files mapper.py,reducer.py \
  -mapper 'python3 mapper.py' \
  -reducer 'python3 reducer.py' \
  -input input/input.txt \
  -output output/wordcount

hdfs dfs -ls -h output/wordcount
hdfs dfs -cat output/wordcount/part-00000

Common Pitfalls

  • Changing dfs.blocksize does not merge existing files. It only affects how new files are divided into blocks.

  • Parquet, ORC, and compression are not complete solutions. They improve read efficiency but do not eliminate excessive HDFS file metadata.

  • Oversized compacted files can hurt parallelism. One enormous file may serialize processing that should run concurrently.

  • Compaction must be validated before old data is removed. Write new output, compare counts or checksums, and publish it safely.

  • Erasure coding does not solve small files. It reduces replication storage overhead but still operates on files and blocks with associated metadata.

Interviewers commonly follow up by asking how you would identify affected directories, choose compaction file sizes, avoid data loss, and balance compaction cost against downstream parallelism.

45

How would you recover a Hadoop cluster after a NameNode failure?

Recovering after a NameNode failure normally means invoking HDFS High Availability (HA): ZooKeeper-driven fencing removes the failed active, a standby becomes active, and DataNodes retain the data blocks. The main trade-off is lower recovery time and metadata-loss risk versus the operational complexity of shared edits, fencing, and a healthy ZooKeeper ensemble.

Read full answer

Applies toHadoop 3.x with HDFS High Availability (Python 3.12 for the Streaming example)

HDFS HA Recovery

HA requires a JournalNode quorum, ZooKeeper, a ZKFC process on each NameNode, a fencing method, and shared edit logs.

  1. Detect failure: The ZKFC health monitor detects that the active NameNode is unavailable or has lost its ZooKeeper session.

  2. Fence the old active: Prevent the failed NameNode from continuing to write metadata. This avoids split-brain state and is mandatory before another NameNode becomes active.

  3. Promote a standby: One standby becomes active. Hadoop 3.x supports multiple standby NameNodes, although only one can be active. The standby normally already has current namespace state from its checkpoint and shared edits; a restarted or stale standby must catch up before promotion.

  4. Restore block awareness: The new active receives registrations and block reports from the DataNodes. It does not need to copy data blocks between them.

  5. Validate HDFS: Check for missing, corrupt, and under-replicated blocks. HDFS can create another replica from a surviving copy; if every copy is gone, the block is unrecoverable data loss.

  6. Restore redundancy: Repair the failed NameNode and restart it as a standby so it catches up with the shared edit log before it can be considered fully recovered.

Useful checks include:

hdfs dfsadmin -report
hdfs fsck / -list-corruptfileblocks

Recovery Without HA

If no standby exists, reconstruct and restore the metadata:

  • Install a NameNode with the same cluster identity, compatible configuration, and compatible metadata format.

  • Restore the latest fsimage checkpoint and apply the required edit logs in the correct sequence.

  • Preserve the namespace, ownership, permissions, encryption-zone metadata, and other information contained in the image.

  • Start the NameNode after validating the metadata, then allow the existing DataNodes to reconnect and report their blocks.

  • Use hdfs namenode -recover only for supported metadata-corruption scenarios; it is not a routine substitute for HA or tested backups.

Core Components and Fault Tolerance

  • HDFS: The NameNode manages metadata, while DataNodes store blocks. The default block size is 128 MB and the default replication factor is 3. Hadoop 3.x also supports erasure coding for more storage-efficient durability.

  • YARN: The ResourceManager allocates resources, NodeManagers host containers, and each ApplicationMaster supervises one job. Failed tasks can be re-executed within the configured retry limit.

  • MapReduce: For word count, the mapper emits (word, 1) pairs. Shuffling and sorting place each key together, reducers sum the counts, optional combiners pre-aggregate mapper output, and partitioners determine which reducer receives each key.

  • Data locality: YARN uses the locations reported for HDFS blocks to schedule containers near their data when possible, reducing network traffic.

  • Small files: Many tiny files consume excessive NameNode metadata and scheduler resources, slowing recovery and job startup.

Ecosystem and Current Role

Component Role
Hive SQL-based data warehousing on Hadoop
HBase Distributed wide-column NoSQL database
Pig Legacy data-flow language
Sqoop Legacy bulk import/export tool; retired Apache project
Oozie Hadoop workflow scheduling
ZooKeeper Coordination, leader election, and failover support
Spark Faster in-memory engine that often reads HDFS and runs on YARN

Many teams now use cloud object storage such as S3, GCS, or ADLS, together with Spark and cloud warehouses. HDFS and YARN concepts nevertheless remain common interview topics because distributed storage, data locality, and fault tolerance still matter.

Code Example

The following Hadoop Streaming example validates HDFS I/O and MapReduce execution after recovery. It assumes python3.12 is installed and available on every task-execution node.

# mapper.py
import sys

for line in sys.stdin:
    for word in line.split():
        sys.stdout.write(f"{word}\t1\n")
# reducer.py
import sys

current_word = None
total = 0

for line in sys.stdin:
    word, count_text = line.rstrip("\n").split("\t", 1)
    count = int(count_text)

    if current_word is None:
        current_word = word
        total = count
    elif word == current_word:
        total += count
    else:
        sys.stdout.write(f"{current_word}\t{total}\n")
        current_word = word
        total = count

if current_word is not None:
    sys.stdout.write(f"{current_word}\t{total}\n")

Upload the scripts and input, then run the job:

printf 'hadoop hdfs spark\nhadoop hdfs\n' > input.txt

hdfs dfs -mkdir -p /tmp/wordcount/input
hdfs dfs -put input.txt /tmp/wordcount/input/input.txt
hdfs dfs -ls /tmp/wordcount/input

hadoop jar "${HADOOP_HOME}/share/hadoop/tools/lib/hadoop-streaming-"*.jar \
  -input /tmp/wordcount/input \
  -output /tmp/wordcount/output \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -files mapper.py,reducer.py

hdfs dfs -ls /tmp/wordcount/output
hdfs dfs -cat '/tmp/wordcount/output/part-*'

The output directory must not already exist. The part-* files contain tab-separated words and counts.

Common Pitfalls

  • Untested failover: Regularly test automatic promotion, fencing, ZooKeeper coordination, and JournalNode quorum behavior.

  • Ignoring backups: HA does not protect against simultaneous loss of all NameNode and JournalNode metadata.

  • Assuming all missing blocks can be replicated: At least one surviving replica is required.

  • Leaving the old active out of service: Restart it as a standby to restore normal HA redundancy.

  • Deleting DataNodes unnecessarily: They contain the actual blocks and should remain connected when their data is valid.

46

What are Hadoop backup and disaster recovery considerations?

Hadoop disaster recovery (DR) combines backups, replication, and failover to restore HDFS data and dependent services after component failure, corruption, accidental deletion, or site loss. The main trade-off is faster recovery and less data loss versus additional storage and bandwidth, operational complexity, and consistency risks.

Read full answer

Applies toApache Hadoop 3.x

Recovery Objectives

  • RPO (Recovery Point Objective): The maximum tolerable data loss, expressed as time. A 15-minute RPO permits the previous 15 minutes of changes to be lost.

  • RTO (Recovery Time Objective): The maximum acceptable time to restore service. A low RTO may require automated failover rather than a manual restore.

  • Scope and priorities: Design separately for DataNode, NameNode, corruption, deletion, ransomware, application, and site failures. Prioritize datasets and services according to business impact.

  • Consistency: Define the recovery point and whether asynchronous replication can introduce stale data or conflicts.

Protect HDFS Data

  • Replication: HDFS normally stores each 128 MB block on three DataNodes. Replication handles disk and node failures, but deletion can propagate to every replica and provides no historical version.

  • Erasure coding: Hadoop 3.x erasure coding reduces overhead compared with three-way replication. Reconstruction uses more network and disk I/O, and erasure coding is not a versioned backup.

  • Snapshots: HDFS snapshots preserve point-in-time content for configured directories. Recover by copying files from .snapshot or cloning a snapshot; snapshots remain until explicitly deleted.

  • Bulk copies: Use distcp for HDFS-to-HDFS copies, using a source snapshot when a consistent copy point is required.

  • Off-site copies: Keep copies in another failure domain, such as S3, GCS, ADLS, or another cluster. Use immutable retention where available.

  • Small files: Avoid backups dominated by tiny files. Compaction or suitable packaging can reduce NameNode metadata load and restore time.

Protect Metadata and Services

  • NameNode metadata: Checkpoint the namespace and copy the resulting fsimage and required edit logs off-site. A CheckpointNode can create checkpoints, but it is not itself an off-site backup.

  • NameNode HA: Hadoop 3.x supports multiple standby NameNodes and shared JournalNode edits, with only one Active NameNode. HA handles component failure but does not replace independent metadata backups for site loss.

  • Hive and HBase: Back up the Hive Metastore database alongside matching HDFS table data. For HBase, follow documented snapshot, WAL, and restore procedures rather than copying live HDFS files.

  • Coordination and security: Protect ZooKeeper ensembles and configuration, Kerberos keytabs, ACLs, and actual KMS/HSM key-recovery material. Configuration alone cannot restore encrypted data.

  • YARN and MapReduce: Back up relevant ResourceManager, NodeManager, and ApplicationMaster configuration. MapReduce re-executes failed tasks, but recovery still depends on durable HDFS inputs and committed outputs.

Select a DR Architecture

Mechanism Protects against Important limitation
HDFS replication Disk or node failure No history or deletion protection
HDFS snapshots Accidental changes Limited to configured paths
distcp or object copy Cluster or site loss Consistency and lag affect RPO
NameNode HA Active NameNode failure Does not protect site loss by itself
Erasure coding Storage overhead Slower reconstruction; no versions
Cross-site replication Site failure Requires dedicated tooling and conflict policy
  • Active-passive: Maintain a secondary cluster and fail over after site loss. This is simpler but may expose stale data.

  • Pilot-light: Keep configuration and metadata ready while reproducing data during recovery. This lowers cost but usually increases RTO.

  • Active-active: Native HDFS HA is single-writer. Cross-site active-active requires a multi-writer filesystem or synchronization layer with explicit conflict semantics.

The runbook should cover detection, fencing, promotion, client redirection, capacity checks, service startup order, security reconfiguration, and rollback.

Verify Recovery

  • Restore regularly into an isolated environment; a successful backup job does not prove the backup is usable.

  • Validate checksums, file counts, schemas, HBase tables, and expected business records.

  • Exercise both NameNode failover and full-site recovery, measuring actual RPO and RTO.

  • Monitor snapshot success, backup age, replication lag, checkpoint completion, and free capacity.

  • Keep off-site backups isolated from production and administrator credentials, using immutable retention where possible.

  • Many teams now combine cloud object storage with Spark or cloud warehouses, while Spark also commonly runs on YARN and reads HDFS. HDFS/YARN data locality, fault tolerance, and tested recovery remain important interview topics.

Common Pitfalls

  • Treating three HDFS replicas as a versioned backup.

  • Assuming NameNode HA protects against complete data-center loss.

  • Backing up NameNode metadata but omitting Hive, HBase, ZooKeeper, and encryption-key recovery.

  • Copying live data without defining a consistent recovery point.

  • Never testing restoration or ignoring small-file and capacity effects.

  • Designing a new pipeline around retired components such as Sqoop instead of supported ingestion and replication tools.

47

How do you monitor Hadoop cluster health?

Hadoop cluster monitoring combines HDFS, YARN, MapReduce, and infrastructure health signals to detect capacity exhaustion, node failures, data loss, and performance degradation before users are affected. A common stack uses Prometheus, JMX Exporter, Grafana, and Alertmanager, supplemented by Hadoop CLI diagnostics and centralized logs. The main trade-off is monitoring overhead: detailed metrics and alerts require tuning, but insufficient monitoring can hide failed redundancy, unhealthy nodes, or overloaded workloads.

Read full answer

Applies toHadoop 3.x (examples in Python 3.12)

Core Health Signals

  • HDFS and NameNode

  • Track filesystem capacity, remaining space, safemode state, metadata operation latency, and NameNode failover status.

  • Hadoop 3.x supports multiple standby NameNodes. Automatic failover uses ZooKeeper coordination; production deployments must also configure a supported fencing mechanism to prevent split-brain scenarios.

  • Monitor live and dead DataNodes, failed volumes, missing blocks, corrupt blocks, and under-replicated blocks.

  • The default HDFS block size is 128 MB and the default replication factor is 3. Erasure coding is an optional alternative, so alerts must account for the active storage policy.

  • YARN

  • Monitor ResourceManager queue capacity, NodeManager health, container failures, ApplicationMaster state, and node losses.

  • Investigate long queues, excessive container churn, and resources allocated to a single queue or tenant.

  • MapReduce

  • Track map and reduce progress, failed and killed attempts, speculative execution, spill volume, shuffle time, and data skew.

  • Retried tasks provide fault tolerance, but repeated failures indicate an unhealthy node, overloaded cluster, or problematic input.

  • Infrastructure

  • Collect CPU, memory, disk latency, network throughput, load, and filesystem inode usage from every host.

  • Data locality should normally keep computation close to HDFS blocks; widespread remote reads indicate scheduling or cluster-balance problems.

  • Ecosystem services

  • Monitor Hive Metastore, HBase, ZooKeeper, and Spark workloads. Spark commonly runs on YARN and reads from HDFS.

  • Pig is legacy, Sqoop is retired, and Oozie provides workflow scheduling, so their operational importance varies by installation.

Recommended Tools

Tool or layer How it works Strengths Weaknesses Typical use
Prometheus and JMX Exporter JMX Exporter exposes Hadoop and JVM JMX metrics in Prometheus format; Prometheus scrapes and stores them Detailed alerts and historical analysis Requires configuration and metric storage Continuous, cluster-wide monitoring
Grafana and Alertmanager Visualizes dashboards and routes alerts Fast diagnosis and team integration Dashboards can become noisy without thresholds Operational visibility and alerting
Hadoop CLI diagnostics Queries HDFS and YARN directly Authoritative, low setup cost Manual and unsuitable as the only monitoring Incident investigation and health checks
Centralized logging Collects YARN aggregated logs and host logs with OpenSearch or similar tools Good for distributed job diagnosis Storage and indexing costs grow with log volume Root-cause analysis
Vendor management platforms Provide product-specific health models and support Integrated support and guidance Licensing and vendor dependency Managed enterprise clusters

Practical Monitoring Workflow

  1. Establish baselines for storage capacity, block redundancy, queue utilization, task duration, and failure rates.

  2. Create alerts for missing blocks, exhausted capacity, dead DataNodes, repeated task failures, and abnormal NameNode leadership changes.

  3. Use dashboards to identify whether an incident originates in HDFS, YARN scheduling, a particular application, or the underlying hosts.

  4. Run periodic integrity and recovery checks, including hdfs fsck, and verify backups independently of HDFS replication.

  5. Review capacity trends and the small-files problem, which increases NameNode metadata pressure and scheduler overhead.

Code Example

Use Hadoop CLI commands for an immediate health check:

# HDFS capacity, DataNode status, and block redundancy
hdfs dfsadmin -report

# Filesystem integrity; may be expensive on a very large root directory
hdfs fsck / -files -blocks -locations

# YARN NodeManager status
yarn node -list -all

A Hadoop Streaming job can validate the execution path. The mapper emits tab-separated word counts, and the reducer sums them:

# mapper.py
import sys

for line in sys.stdin:
    for word in line.strip().split():
        print(f"{word}\t1")
# reducer.py
import sys

current_word = None
total = 0


def emit() -> None:
    if current_word is not None:
        print(f"{current_word}\t{total}")


for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    count = int(count)

    if word == current_word:
        total += count
    else:
        emit()
        current_word = word
        total = count

emit()

Store and execute the example. Python 3.12 must be installed and available on the PATH of the NodeManager containers:

set -euo pipefail

printf 'hadoop hdfs\nhadoop hdfs\nmapreduce\n' > input.txt

hdfs dfs -mkdir -p /tmp/wordcount/input
hdfs dfs -put input.txt /tmp/wordcount/input/
hdfs dfs -ls /tmp/wordcount/input

STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/mapreduce" \
  -type f -regextype posix-extended \
  -regex '.*/hadoop-streaming-[0-9]+(\.[0-9]+)*\.jar' -print -quit)"
test -n "$STREAMING_JAR"

OUTPUT_PATH="/tmp/wordcount/output-$(date +%s)-$$"

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -input /tmp/wordcount/input \
  -output "$OUTPUT_PATH"

hdfs dfs -ls "$OUTPUT_PATH"
# Expected reducer counts: hadoop 2, hdfs 2, mapreduce 1

Common Pitfalls

  • Monitoring only average resource usage can hide a single hot disk, queue, or tenant.

  • Treating every NameNode failover as an outage ignores expected high-availability behavior; monitor service continuity and repeated failovers.

  • Using HDFS replication as the only backup is unsafe because replication does not protect against logical corruption or accidental deletion.

  • Generating too many low-value alerts causes fatigue; base warnings on impact, trends, and recovery time.

  • Many organizations now use S3, GCS, ADLS, cloud warehouses, and Spark instead of HDFS/YARN. Nevertheless, distributed storage, fault tolerance, data locality, and scheduling remain important interview topics.

48

How do you plan and scale a Hadoop cluster?

Hadoop cluster planning sizes storage, compute, memory, network, and operational headroom from measured workloads, then scales each layer while preserving fault tolerance and data locality. The main trade-off is additional infrastructure cost and operational complexity against better performance, resilience, and growth capacity.

Read full answer

Applies toHadoop 3.x with Python 3.12 for Hadoop Streaming examples

Capacity Planning Process

Plan from representative workloads rather than total data volume alone.

  1. Measure the workload: Record ingestion rate, retention, average and peak file sizes, compression, concurrency, and growth.

  2. Benchmark real jobs: Test representative Hive, MapReduce, HBase, and Spark workloads with production-like volume, skew, and failure conditions.

  3. Size each layer:

  4. Raw disk capacity ≈ retained logical bytes × replication or erasure-coding overhead + temporary space and operating reserve.

  5. Worker nodes ≈ peak CPU, memory, and I/O demand ÷ allocatable resources per node, plus failure and maintenance reserve.

  6. Watch bottlenecks: Track disk latency, network throughput, shuffle volume, queue wait time, container locality, failed attempts, and NameNode metadata pressure.

Scaling approach Best use Main trade-off
Add DataNodes HDFS capacity or storage throughput Requires rebalancing and network capacity
Add NodeManagers Concurrent YARN containers Increases contention and operational overhead
Enlarge nodes A specific CPU, memory, or disk bottleneck Requires hardware replacement
Change storage policy Lower storage overhead EC saves space but costs more CPU and network
Move compatible workloads Batch analytics not requiring HDFS/YARN Adds copying, governance, and operational work

HDFS Capacity and Fault Tolerance

The NameNode stores namespace and block metadata, while DataNodes store the blocks. Hadoop 3.x defaults to a 128 MB block size and replication factor 3.

  • Replication: Plan for three copies, preferably across racks. Three replicas on distinct DataNodes can tolerate two node failures, although capacity and placement must still be verified.

  • Erasure coding: Hadoop 3.x can reduce storage overhead by encoding groups of blocks. It generally increases CPU, network, and recovery overhead.

  • NameNode high availability: Hadoop 3.x supports multiple standby NameNodes. Size the NameNode heap for the number of files and blocks, and provision the JournalNode quorum used by HDFS HA.

  • Data locality: Schedule computation near the required HDFS blocks. Remote reads can dominate runtime when locality is poor.

  • Small files: Many small files burden NameNode metadata and increase scheduling and I/O overhead. Compact them or use formats suited to small-file workloads.

  • Fault tolerance: DataNode replication protects stored data, and YARN re-executes failed tasks, commonly on another node after failure or retry exhaustion.

YARN and MapReduce Planning

YARN separates resource management from processing:

  • ResourceManager: Allocates resources among queues and applications.

  • NodeManager: Reports and manages resources on each worker node.

  • ApplicationMaster: Coordinates an application and requests containers from the ResourceManager.

Size CPU and memory at the container level. Account for NodeManager reservations, operating-system overhead, executor or JVM overhead, shuffle buffers, skew, and temporary failure headroom.

MapReduce word count follows these steps:

  1. The mapper emits (word, 1) for each token.

  2. A combiner performs local partial summation when the operation is associative and commutative.

  3. The partitioner selects the reducer for each key.

  4. The shuffle and sort phase groups keys and transfers them to reducers.

  5. The reducer sums the values for each word.

Ecosystem and Modern Alternatives

  • Hive: SQL-based analytics over Hadoop data.

  • HBase: Distributed wide-column database for random row-level access.

  • Pig: Legacy data-flow language, uncommon in new designs.

  • Sqoop: Retired Apache project formerly used for relational-data imports.

  • Oozie: Hadoop workflow scheduling.

  • ZooKeeper: Distributed coordination and synchronization.

  • Spark: Faster, primarily in-memory engine that often runs on YARN and reads HDFS.

Many teams now use S3, GCS, ADLS, Spark, or cloud warehouses for compatible workloads. HDFS/YARN concepts remain important because distributed storage, locality, fault tolerance, resource isolation, and capacity economics are still core interview topics.

Code Example

Save the following as mapper.py. It emits whitespace-delimited words and counts:

import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

Save this as reducer.py. Hadoop Streaming sorts reducer input by key, so the reducer can aggregate incrementally without retaining the entire vocabulary:

import sys

current_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t", 1)
    if current_word is not None and word != current_word:
        print(f"{current_word}\t{count}")
        count = 0
    current_word = word
    count += int(value)

if current_word is not None:
    print(f"{current_word}\t{count}")

Run these commands from the directory containing both scripts. Python 3.12 must be installed on every worker node that can run the containers:

printf 'hadoop hdfs yarn\nhadoop yarn mapreduce\n' > input.txt

export HADOOP_HOME="${HADOOP_HOME:?Set HADOOP_HOME}"
export PATH="$HADOOP_HOME/bin:$PATH"

hdfs dfs -mkdir -p /tmp/interview/input
hdfs dfs -put input.txt /tmp/interview/input/
hdfs dfs -ls -h /tmp/interview/input

STREAMING_JAR="$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -name 'hadoop-streaming-*.jar' -print -quit)"
test -n "$STREAMING_JAR"

OUTPUT="/tmp/interview/wordcount-$(date +%s)"
hadoop jar "$STREAMING_JAR" \
  -input /tmp/interview/input \
  -output "$OUTPUT" \
  -files mapper.py,reducer.py \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 1

hdfs dfs -cat "$OUTPUT"/part-*

Common Pitfalls

  • Using averages only: Peak concurrency and skewed partitions often determine capacity.

  • Ignoring shuffle traffic: Network bandwidth can become the bottleneck before CPU does.

  • Underestimating metadata: Small files can exhaust NameNode resources despite available disk space.

  • Leaving no failure headroom: Maintenance, retries, and rebalancing reduce usable capacity.

  • Oversubscribing CPU: More vcores than physical cores can improve utilization but increase contention.

  • Migration gaps: Moving to object storage can require format changes, data copies, access controls, and consistent testing.

  • Interview follow-ups: How would you calculate replication overhead, handle a DataNode failure, choose replication versus erasure coding, rebalance HDFS, diagnose a slow shuffle, or migrate workloads from HDFS to object storage?

49

How do Hadoop and Spark support real-time clickstream processing?

Real-time clickstream processing in Hadoop typically separates durable capture and replay from low-latency stream processing: HDFS stores immutable events and checkpoints, YARN hosts Spark Structured Streaming, and a broker such as Kafka feeds events to the stream. The main trade-off is low-latency analytics versus the operational cost of checkpoints, state management, ordering, deduplication, and sink consistency.

Read full answer

Applies toApache Hadoop 3.x and Apache Spark 4.0.x

Integration Architecture

  • HDFS: The NameNode manages metadata and DataNodes store blocks. Hadoop 3.x defaults to 128 MB blocks and replication factor 3; erasure coding can replace replication for suitable cold data. A standard HA deployment has one active and one standby NameNode backed by a JournalNode quorum.

  • YARN: The ResourceManager allocates resources, NodeManagers host containers, and each application receives an ApplicationMaster.

  • Spark: Spark executors run on YARN and use HDFS for raw data, checkpoints, and historical outputs. Data locality reduces network movement.

  • Ecosystem: Hive provides SQL analytics over HDFS, HBase supports random-access lookups, and ZooKeeper coordinates distributed services. Pig is legacy, Sqoop is retired, and Oozie handles Hadoop workflow scheduling.

  • MapReduce: It remains useful for batch ETL, reconciliation, and replay, but job startup and shuffle overhead make it unsuitable for continuously updated dashboards.

Real-Time Processing Flow

  1. Clients publish click events to Kafka or another durable broker.

  2. Spark Structured Streaming consumes the events and parses their payloads.

  3. Spark performs validation, deduplication, enrichment, and sessionization.

  4. Event-time windows and watermarks calculate metrics such as clicks per page, campaign, or user.

  5. Spark writes checkpoints to HDFS and sends results to HDFS, HBase, Kafka, or another serving system.

  6. Batch jobs reconcile results and build historical aggregates.

Structured Streaming provides near-real-time processing, not hard real-time guarantees. Exactly-once behavior depends on coordinated checkpointing, replayable sources, and an idempotent or transactionally compatible sink.

Component Responsibilities

Component Role Strength Limitation
HDFS Durable events, checkpoints, and history Fault-tolerant block storage Not an event queue; small files hurt performance
YARN Resource allocation Shared cluster management Requires capacity and queue planning
Spark Structured Streaming Stateful stream processing Low latency and expressive transformations Checkpoints and state require careful management
MapReduce Batch ETL and replay Mature fault-tolerant model High latency for continuous workloads
Hive and HBase Analytics and operational lookups SQL and random-access serving Separate operational and data-model requirements

Code Example

The Hadoop Streaming example establishes a batch word-count baseline:

# mapper.py
import sys

for line in sys.stdin:
    for word in line.lower().split():
        print(f"{word}\t1")
# reducer.py
import sys

current_word = None
total = 0

for line in sys.stdin:
    word, count = line.rstrip("\n").split("\t", 1)
    if word != current_word:
        if current_word is not None:
            print(f"{current_word}\t{total}")
        current_word = word
        total = 0
    total += int(count)

if current_word is not None:
    print(f"{current_word}\t{total}")
cat > clicks.txt <<'EOF'
click landing
click product
signup
EOF

hdfs dfs -mkdir -p /clickstream/input
hdfs dfs -put -f clicks.txt /clickstream/input/clicks.txt
hdfs dfs -ls /clickstream/input

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' | head -n 1)

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -input /clickstream/input/clicks.txt \
  -output /clickstream/output-wordcount \
  -mapper "python3.12 mapper.py" \
  -reducer "python3.12 reducer.py" \
  -numReduceTasks 1

hdfs dfs -cat '/clickstream/output-wordcount/part-*' | sort -k1,1
# click    2
# landing  1
# product  1
# signup   1

This Spark example calculates windowed click counts from a self-contained synthetic source:

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, watermark, window

spark = (
    SparkSession.builder
    .appName("clickstream-demo")
    .config("spark.sql.shuffle.partitions", "2")
    .getOrCreate()
)

events = spark.readStream.format("rate").load()
clicks = (
    events
    .withColumn("page_id", (col("value") % 3).cast("int"))
    .withColumn(
        "event_time",
        watermark(col("timestamp"), "30 seconds"),
    )
)
counts = clicks.groupBy(
    window(col("event_time"), "10 seconds").alias("event_window"),
    "page_id",
).count()

query = (
    counts.writeStream
    .format("memory")
    .queryName("click_counts")
    .outputMode("update")
    .trigger(processingTime="5 seconds")
    .start()
)

try:
    query.awaitTermination(20)
    spark.sql("SELECT * FROM click_counts").show(truncate=False)
finally:
    query.stop()
    spark.stop()

In production, replace the rate source with Kafka and use a durable, idempotent output sink.

Operational Best Practices

  • Use a broker for active delivery and retain immutable raw events in HDFS for replay.

  • Choose event-time windows, watermarks, and late-event policies from reporting requirements.

  • Compact small files and monitor ingestion lag, state growth, executor failures, and HDFS capacity.

  • Use HDFS replication or erasure coding deliberately; YARN re-executes failed tasks, while Spark recovers executors from checkpointed offsets and state stores.

  • Evaluate S3, GCS, ADLS, managed streams, and cloud warehouses alongside Hadoop, but HDFS/YARN concepts remain important interview topics.

Common Pitfalls

  • Treating HDFS as a message queue or writing every event as a separate HDFS file.

  • Applying a watermark to one time column while aggregating another.

  • Claiming exactly-once delivery merely because checkpointing is enabled.

  • Keeping only streaming checkpoints instead of replayable raw events.

  • Ignoring skewed event keys, burst traffic, backpressure, or NameNode high availability.

50

How can Hadoop components support an IoT sensor-data pipeline?

A Hadoop IoT pipeline uses HDFS as a durable landing zone, YARN to execute processing, and Hive, HBase, or Spark for analysis and serving. HDFS replication, distributed execution, and data locality provide fault tolerance and horizontal scalability, but the architecture is batch-oriented rather than optimized for millisecond event delivery. Low-latency ingestion therefore usually requires a gateway or external streaming buffer.

Read full answer

Applies toHadoop 3.x (Python 3.12 examples)

Proposed Pipeline

  1. Ingest: Devices publish readings to a gateway or message buffer. The gateway writes time-partitioned, compressed files to HDFS rather than creating one small file per reading.

  2. Validate and transform: MapReduce or Spark jobs running on YARN remove malformed records, standardize units, attach device metadata, and calculate windowed aggregates.

  3. Curate: Write clean data in Parquet or ORC for efficient analytical scans. Partition by date and optionally by site or device type.

  4. Analyze: Create Hive external tables over curated files for historical SQL analysis, feature generation, and reporting.

  5. Serve recent state: Load current device state into HBase when low-latency lookups are required. Use bulk loading for batch-derived data and normal mutations for near-real-time updates.

  6. Orchestrate: Oozie can schedule dependent Hadoop jobs, although teams may use an external orchestrator for newer workflows.

Component Fit

Component How it works Strengths Weaknesses Typical use or scale
HDFS NameNode metadata and DataNode blocks Durable, fault-tolerant, sequential throughput Small-file and metadata overhead Large datasets across horizontally scaled DataNodes
YARN ResourceManager, NodeManagers, and per-application ApplicationMasters Shared cluster resource management Configuration and scheduling complexity Concurrent batch and analytics workloads
MapReduce Map, shuffle, sort, reduce phases Simple, reliable, data-local Higher latency than in-memory engines Batch processing with linear data-volume cost
Hive SQL compiled into execution plans Familiar analytics and extensibility Query latency depends on the engine Large historical analytical datasets
HBase Wide-column distributed NoSQL store Low-latency random access Operational complexity Large current-state or time-series lookup tables
Spark In-memory distributed transformations Faster iterative and mixed workloads Memory and cluster overhead Stream-like micro-batches and complex analytics

MapReduce Flow

For a word-count-style analysis:

  1. Map: Each mapper reads local input and emits a (word, 1) pair.

  2. Combiner: An optional mapper-side preaggregation reduces shuffled data.

  3. Partitioner: A hash partitioner routes equal keys to the same reducer, preserving grouping.

  4. Shuffle and sort: YARN moves intermediate partitions to reducers and sorts them by key.

  5. Reduce: Each reducer totals values for its keys. Scheduling places mappers near their HDFS blocks, reducing network transfers.

Code Example

mapper.py:

#!/usr/bin/env python3
import sys

for line in sys.stdin:
    for word in line.split():
        print(f"{word}\t1")

reducer.py:

#!/usr/bin/env python3
import sys

last_word = None
count = 0

for line in sys.stdin:
    word, value = line.rstrip("\n").split("\t")

    if last_word is not None and word != last_word:
        print(f"{last_word}\t{count}")
        count = 0

    last_word = word
    count += int(value)

if last_word is not None:
    print(f"{last_word}\t{count}")

Create sample data, upload it, run Hadoop Streaming, and inspect the output:

printf 'device-1 status=ok\ndevice-2 status=warn\ndevice-3 status=ok\n' > sensor.txt

STREAMING_JAR=$(find "$HADOOP_HOME/share/hadoop/tools/lib" \
  -maxdepth 1 -name 'hadoop-streaming-*.jar' -print -quit)

INPUT="/user/$USER/iot/input"
OUTPUT="/user/$USER/iot/output-$(date +%s)"

hadoop fs -mkdir -p "$INPUT"
hadoop fs -put -f sensor.txt "$INPUT/sensor.txt"
hadoop fs -ls -h "$INPUT"

hadoop jar "$STREAMING_JAR" \
  -files mapper.py,reducer.py \
  -mapper "python3 mapper.py" \
  -reducer "python3 reducer.py" \
  -input "$INPUT/sensor.txt" \
  -output "$OUTPUT"

hadoop fs -cat "$OUTPUT"/part-r-*

The cluster’s container environment must provide Python 3.12.

Fault Tolerance and Modern Role

  • HDFS defaults: Blocks are normally 128 MB with replication factor 3. DataNode failures are tolerated through replicas; failed YARN tasks can be re-executed.

  • NameNode HA: Hadoop 3.x supports multiple standby NameNodes with shared edit storage, fencing, and automatic failover. Erasure coding can reduce storage overhead for cold data, although reconstruction consumes more CPU and network.

  • Ecosystem placement: Pig is legacy batch-processing technology; Sqoop is retired and should not be selected for new pipelines. Oozie handles Hadoop workflows, while ZooKeeper provides distributed coordination.

  • Current deployments: Many teams now use S3, GCS, ADLS, Spark, and cloud warehouses. HDFS/YARN concepts remain important because they explain distributed storage, data locality, fault tolerance, and cluster resource isolation.

Common Pitfalls

  • Writing one HDFS file per event creates the small-files problem and burdens the NameNode.

  • Treating HDFS as a low-latency message queue leads to excessive latency and metadata pressure.

  • Ignoring late, duplicated, or missing readings corrupts time-windowed analytics.

  • Keeping replication factor 3 on every dataset may be unnecessarily expensive; use replication and erasure coding according to access patterns.

  • Rebuilding Hadoop for every workload ignores YARN’s multi-tenant isolation, Hive’s SQL reuse, and Spark’s faster iterative processing.

Sign in to keep reading

You've opened 2 full answers. Sign in to read all 50 Hadoop answers, mark what you know and pick up where you left off.

Report a mistake

What's wrong?

All Machine Learning & Data Science subjects