PySpark Architecture

Apache Spark

• Spark Core is written in Scala - Spark is a computation engine. It runs inside a Java Virtual Machine (JVM).

• Apache Spark is an open-source analytical processing engine for large-scale, powerful distributed data processing and machine learning applications. Spark was originally developed at the University of California, Berkeley, and later donated to the Apache Software Foundation. In February 2014, Spark became a Top-Level Apache Project and has been contributed to by thousands of engineers, making Spark one of the most active open-source projects in Apache.

• The framework supports multiple programming languages, including Scala, Python (PySpark), R (SparkR/SparklyR), and Java.

Storage

(HDFS, S3, Azure Blob, GCS)

        │

        ▼

     Apache Spark

 (Reads → Processes → Writes)

        │

        ▼

Output

(Database, Data Lake, Kafka, Files)

Spark performs distributed computation, not distributed storage.

• Driver Program — The "brain": Runs the main() function, creates SparkSession/SparkContext, translates code into a logical plan, optimizes it (via Catalyst), creates a physical plan, and coordinates execution.

• Cluster Manager — Allocates resources (YARN, Kubernetes, Mesos, standalone, or cloud-native like Databricks/EMR).

• Executors — Worker processes on cluster nodes. Each runs tasks, stores data in memory/disk, and reports back to the driver.

• SparkSession — Unified entry point (replaces older SparkContext + SQLContext).

• Spark Connect (major in 4.x) — Decouples client from cluster: Thin client (Python/Scala/Go/R) talks to server-side Spark via gRPC. Enables remote development, better security, and consistency between local and cluster execution.

PySpark Architecture:

• Cluster Manager: Coordinates resource allocation and task scheduling across the cluster nodes.

• Driver Program: Maintains the SparkContext/SparkSession, manages the execution environment, and converts user code into a Directed Acyclic Graph (DAG) of transformations.

• Executors: Worker processes responsible for executing tasks and storing data in-memory or on disk. Tasks run within the Executor's JVM.
  
  Spark Core

Responsible for

• Scheduling

• Memory Management

• Fault Recovery

• Task Distribution

• RDD API

Everything depends on Spark Core. It contains several modules.            

Structured Streaming

Processes streaming data continuously.

Example:  Kafka → Spark → Delta Table

Key interview takeaways

• Spark is a distributed computation engine, not a storage system.

• The Driver plans execution; Executors perform the computation.

• The Cluster Manager allocates resources but does not execute user code.

• An Application contains one Driver and one or more Executors.

• An Action triggers a Job.

• A Job is divided into one or more Stages.

• A Stage is divided into Tasks.

• One Task processes one Partition.

• Narrow transformations do not require shuffles and stay within the same stage.

• Wide transformations require shuffles and create new stages.

• Spark uses lazy evaluation and builds a DAG before execution.

• RDDs are immutable and fault tolerant through lineage rather than replication.

                     

       

Spark Execution Model

Spark applications follow a hierarchical execution structure:

• Job: The highest level of execution, triggered by an action (e.g., collect(), save()).

• Stage: A set of tasks that can be executed in parallel. Stages are divided based on shuffle boundaries created by wide transformations.

• Task: The smallest unit of work, sent to an executor to be performed on a single data partition.

Components:

1- Driver Program

The Driver is the brain. Without a Driver, Nothing executes.

Responsibilities

• Runs your application

• Converts code into execution plan

• Builds DAG

• Creates Jobs

• Divides Jobs into Stages

• Divides Stages into Tasks

• Sends Tasks to Executors

• Receives results

> df = spark.read.csv("employees.csv")

> df.filter(df.salary > 50000).groupBy("dept").count().show()

Entire code first executes on Driver. The driver never processes data itself. Instead it creates an execution plan.
Driver contains SparkSession (Entry point of Spark).

SparkSession internally creates

• SparkContext

• SQLContext

• HiveContext

2- Cluster Manager

Driver needs machines. Who provides them? Cluster Manager.

Responsibilities

• Allocate CPUs

• Allocate Memory

• Launch Executors

• Monitor Resources

Spark supports multiple cluster managers. Standalone(Single machine)  , YARN(Hadoop storage and nodes), Kubernetes (AWS K8s or Manages K8s Master and worker node Architecture)

3- Executors

Executors perform actual computation. 

Contains

• Task Slots

• Memory

• Cache

• Shuffle Files

Executor responsibilities

• Execute tasks

• Cache Data

• Return results

• Write shuffle output

Executors never communicate directly with Driver except for reporting results and status.

Stages and Transformations

The Spark optimizer pipelines transformations into stages based on data dependencies:

• Narrow Transformations: Operations where each input partition contributes to at most one output partition (e.g., map, filter). These do not require data shuffling across the network and can be executed within a single stage.

• Wide Transformations: Operations requiring data from multiple partitions to be redistributed across the cluster (e.g., groupByKey, join, reduceByKey). These require a shuffle and define the start of a new stage.