Apache Spark
Spark Core is written in Scala - Spark is a computation engine. It runs inside a Java Virtual Machine (JVM).
Apache Spark is an open-source analytical processing engine for large-scale, powerful distributed data processing and machine learning applications. Spark was originally developed at the University of California, Berkeley, and later donated to the Apache Software Foundation. In February 2014, Spark became a Top-Level Apache Project and has been contributed to by thousands of engineers, making Spark one of the most active open-source projects in Apache.
The framework supports multiple programming languages, including Scala, Python (PySpark), R (SparkR/SparklyR), and Java.
Storage
(HDFS, S3, Azure Blob, GCS)
│
▼
Apache Spark
(Reads → Processes → Writes)
│
▼
Output
(Database, Data Lake, Kafka, Files)
Spark performs distributed computation, not distributed storage.
Driver Program — The "brain": Runs the main() function, creates SparkSession/SparkContext, translates code into a logical plan, optimizes it (via Catalyst), creates a physical plan, and coordinates execution.
Cluster Manager — Allocates resources (YARN, Kubernetes, Mesos, standalone, or cloud-native like Databricks/EMR).
Executors — Worker processes on cluster nodes. Each runs tasks, stores data in memory/disk, and reports back to the driver.
SparkSession — Unified entry point (replaces older SparkContext + SQLContext).
Spark Connect (major in 4.x) — Decouples client from cluster: Thin client (Python/Scala/Go/R) talks to server-side Spark via gRPC. Enables remote development, better security, and consistency between local and cluster execution.
PySpark Architecture:
Cluster Manager: Coordinates resource allocation and task scheduling across the cluster nodes.
Driver Program: Maintains the SparkContext/SparkSession, manages the execution environment, and converts user code into a Directed Acyclic Graph (DAG) of transformations.
Executors: Worker processes responsible for executing tasks and storing data in-memory or on disk. Tasks run within the Executor's JVM.
Spark Core
Responsible for
Scheduling
Memory Management
Fault Recovery
Task Distribution
RDD API
Everything depends on Spark Core. It contains several modules.
Structured Streaming
Processes streaming data continuously.
Example: Kafka → Spark → Delta Table
Key interview takeaways
Spark is a distributed computation engine, not a storage system.
The Driver plans execution; Executors perform the computation.
The Cluster Manager allocates resources but does not execute user code.
An Application contains one Driver and one or more Executors.
An Action triggers a Job.
A Job is divided into one or more Stages.
A Stage is divided into Tasks.
One Task processes one Partition.
Narrow transformations do not require shuffles and stay within the same stage.
Wide transformations require shuffles and create new stages.
Spark uses lazy evaluation and builds a DAG before execution.
RDDs are immutable and fault tolerant through lineage rather than replication.
Spark Execution Model
Spark applications follow a hierarchical execution structure:
Job: The highest level of execution, triggered by an action (e.g., collect(), save()).
Stage: A set of tasks that can be executed in parallel. Stages are divided based on shuffle boundaries created by wide transformations.
Task: The smallest unit of work, sent to an executor to be performed on a single data partition.
Components:
1- Driver Program
The Driver is the brain. Without a Driver, Nothing executes.
Responsibilities
Runs your application
Converts code into execution plan
Builds DAG
Creates Jobs
Divides Jobs into Stages
Divides Stages into Tasks
Sends Tasks to Executors
Receives results
> df = spark.read.csv("employees.csv")
> df.filter(df.salary > 50000).groupBy("dept").count().show()
Entire code first executes on Driver. The driver never processes data itself. Instead it creates an execution plan.
Driver contains SparkSession (Entry point of Spark).
SparkSession internally creates
SparkContext
SQLContext
HiveContext
2- Cluster Manager
Driver needs machines. Who provides them? Cluster Manager.
Responsibilities
Allocate CPUs
Allocate Memory
Launch Executors
Monitor Resources
Spark supports multiple cluster managers. Standalone(Single machine) , YARN(Hadoop storage and nodes), Kubernetes (AWS K8s or Manages K8s Master and worker node Architecture)
3- Executors
Executors perform actual computation.
Contains
Task Slots
Memory
Cache
Shuffle Files
Executor responsibilities
Execute tasks
Cache Data
Return results
Write shuffle output
Executors never communicate directly with Driver except for reporting results and status.
Stages and Transformations
The Spark optimizer pipelines transformations into stages based on data dependencies:
Narrow Transformations: Operations where each input partition contributes to at most one output partition (e.g., map, filter). These do not require data shuffling across the network and can be executed within a single stage.
Wide Transformations: Operations requiring data from multiple partitions to be redistributed across the cluster (e.g., groupByKey, join, reduceByKey). These require a shuffle and define the start of a new stage.
