What is Apache Spark and what are its core components?
Apache Spark is an open-source, distributed data processing engine that runs computations in memory across a cluster, making it well suited for large-scale batch analytics, SQL queries, machine learning, and stream processing. It solves the problem of processing datasets that are too large or too slow for a single machine by parallelizing work across many nodes, trading the simplicity of single-machine tools for cluster coordination overhead and operational complexity.
Read full answerHide full answer
Applies toApache Spark 3.5 / 4.x with PySpark (Python 3.12)
Core Components
-
Cluster manager: Allocates resources across the cluster. Spark works with its standalone manager, YARN, Kubernetes, or Mesos; Spark submits an application and requests CPU cores and memory for executors.
-
Driver program: The process that runs the user’s
maincode, creates theSparkSession, builds the logical plan, and coordinates work. It tracks metadata and collects small results when explicitly requested. -
Executors: JVM processes launched on worker nodes that run tasks, store cached data in memory or on disk, and return results to the driver. Each application typically gets its own executors for the duration of the job.
-
Spark SQL and DataFrames: The primary API for structured data. It provides the
DataFrame/Dataset abstraction and a SQL interface, optimized by the Catalyst optimizer and executed by the Tungsten engine for efficient memory and code generation. -
Structured Streaming: A streaming engine built on the same DataFrame/SQL model, processing continuous data as incremental micro-batches or (where supported) continuous processing.
-
MLlib (
pyspark.ml): Spark’s machine learning library using the DataFrame-based Pipeline API (VectorAssembler,StringIndexer, estimators, transformers). The older RDD-basedpyspark.mllibAPI is deprecated. -
GraphX: A graph processing library on Spark; in Python workflows, graph analysis is often handled with external libraries or Spark SQL patterns because GraphX is primarily a Scala/Java API.
Execution Model
-
Lazy transformations vs. actions: Transformations such as
select,filter,join, andgroupBybuild a logical plan but do not run immediately. Actions such ascount,show,write, andcollecttrigger execution. -
DAG, stages, and tasks: Spark builds a directed acyclic graph (DAG) of transformations. The scheduler splits the DAG into stages at shuffle boundaries; each stage is divided into tasks, typically one task per input partition, which run in parallel on executors.
-
Narrow vs. wide transformations: Narrow transformations (for example,
filter,map-like column expressions) do not require data to move across partitions. Wide transformations (for example,groupBy,join,repartition) cause a shuffle, redistributing data across the network and often dominating runtime. -
Optimization and partitioning: Catalyst rewrites queries (predicate pushdown, projection pruning, join reordering), while Adaptive Query Execution (AQE), on by default since Spark 3.2, can adjust shuffle partitions, optimize joins (including broadcast decisions), and mitigate skew at runtime.
repartitionperforms a full shuffle to change partition count;coalescereduces partitions with less movement.
Comparison with Related Tools
| Tool | How it works | Strengths | Weaknesses | When to use |
|---|---|---|---|---|
| Apache Spark | Distributed in-memory engine with DataFrame/SQL APIs | Scales to large clusters, unified batch/SQL/ML/streaming | Cluster overhead, tuning required | Data larger than one machine or multi-node workloads |
| pandas | Single-process, in-memory DataFrames | Simple, rich API, fast iteration on small data | Limited by one machine’s memory/CPU | Data that comfortably fits on one machine |
| Dask | Parallel Python collections, local or distributed | Familiar pandas/NumPy style, flexible scaling | Different semantics/performance trade-offs than Spark | Scaling Python workflows without a full Spark stack |
| Polars | Single-node (with limited streaming/lazy features) columnar engine | Very fast local analytics, efficient memory use | Not a general multi-node cluster engine like Spark | High-performance local DataFrame processing |
| Hadoop MapReduce | Disk-oriented batch Map and Reduce stages | Reliable batch processing on HDFS ecosystems | Much slower iterative workloads, verbose programming model | Legacy Hadoop batch jobs; Spark usually preferred today |
Interview Tips
-
Emphasize the DataFrame API and Spark SQL for modern Spark; mention RDDs only for low-level control or legacy contexts.
-
Common pitfalls: calling
collect()on large data (pulls results to the driver), using Python UDFs where built-inpyspark.sql.functionsor pandas UDFs would be faster, ignoring shuffle and skew, and over-caching without measuring benefit. -
Mention practical ecosystem choices: columnar formats like Parquet, table formats such as Delta Lake or Apache Iceberg, and
df.explain()to inspect physical plans. -
Follow-up questions interviewers often ask: “What causes a shuffle?”, “When would you use
broadcast?”, “How does AQE change execution?”, and “When is Spark overkill compared with pandas or Polars?”