1. Why is Spark processing faster than MapReduce Jobs?
Spark is in memory computation means spark stores the data in ram and accesses it and processes it in memory only whereas mapreduce stores the data in disk and does read and write operation at every stage. Which takes time and it is expensive.
2. Spark vs Mapreduce
Spark - It is a processing engine built on scala and uses many APIs like pyspark or java-api etc. It is a ‘In memory’ {RAM} and distributed computations, data gets distributed based on a combination of nodes and core. This is a resilient computation platform. Spark can use K8s , hadoop, docker cluster for storage and computation.
Mapreduce is data processing engine: used in hadoop framework to utilise the -> it uses YARN-> resource manager, mapping - mapping data in key value pair, Sorting -> sorting data based on key, reducer -> processing data and aggregating and computing and providing final result.
3. Why was Spark Developed ?
It is developed to overcome the limitations of mapreduce : which is to read/write from disk on every operation and storage will happen on disk.
4. What is spark ?
Spark is a computation engine which utilises distributed architecture. It has master and worker architecture. Actual data is processed on nodes only. And all the data will be stored in node RAMs only. Master will utilise the yarn for resource allocation and distribute the data across the nodes and assign resources.
5. What is PySpark?
Pyspark is an API for python on spark.
6. What are the characteristics of PySpark?
It utilises many functionalities for python and spark users.
PySpark first converts in java wrapper then converts in scala to execute on nodes.
7. Features of Spark and Advantages & Disadvantages of pyspark ?
In memory -> faster and efficient , AEQ (adaptive query execution will check for data skew and if partition is larger than RAM then it can be broken down in more chunks to process), pyspark can recreate partitions and help in writing efficient programs.
RAM can have less space than required and then data spill happens which writes extra data in the disk of nodes (this can be an issue : if the partition is larger than the RAM size and data skew is there and the task will keep happening due to less resource and never completing).
8. What is Spark Driver ?
Spark driver is the brain of the spark engine which helps in resource allocation using cluster manager (YARN) and is used for sending tasks and stages to workers. Drive checks if all tasks happen or not: in case of any action : all the data will be aggregated and data transfer will happen on master node or in destination location as required from the user. The driver will maintain all the metadata and check availability of the resources across nodes : it ensures resilient computation power : which results in maintaining required no of cores and ram and no of nodes.
Cluster Manager (YARN, Kubernetes, or Standalone Master) is responsible for allocating cluster resources and launching executors.
Driver is the brain of the Spark application. It creates the DAG, divides it into stages and tasks, schedules tasks on executors, maintains application metadata (such as lineage, stage/task information, and partition metadata), and coordinates execution.
Executors run on worker nodes. They process partitioned data in parallel, cache data in memory when required, perform shuffle operations, spill data to disk if memory is insufficient, and write the final output to storage.
The driver requests resources through the Cluster Manager.
Cluster manager may be
YARN
Kubernetes
Standalone
Driver responsibilities
Creates SparkContext
Creates DAG
Schedules stages
Sends tasks
Collects results
Tracks executors
9. PySpark Architecture ?
Pyspark Architecture is the same as Spark architecture : It has a master and worker node Archi.
Applicationmanager (driver application) has all metadata
The worker has all actual data for processing.
When reading data from source : master never reads the data all the data will directly go in worker nodes.
Main components are
Driver
Cluster Manager
Executors
SparkContext
DAG Scheduler
Task Scheduler
Driver responsibilities
Creates SparkContext
Creates DAG
Schedules stages
Sends tasks
Collects results
Tracks executors
Driver -> SparkContext -> Cluster Manager -> Executors -> Tasks
DAG -> Directed acyclic graph -> Transformations create lineage. Execution DAG is built and submitted when an Action is triggered.
YARN/K8s {Cluster manager}-> help in resource allocation and maintaining the no of nodes
Master Nodes -> keeps the metadata -> no of partition,size of partition, DAG, metadata
Worker Nodes (executors) - > process all the data in-memory and keep it there until not asked to write in some place
10. PySpark Modules & Packages:
Modules are in-built functions which can utilise to efficiently write the spark function
Packages -> needs to install and require to run Pyspark in python env
from pyspark.sql import SparkSession
from pyspark.sql.functions import *
11. Spark Components?
Driver
Cluster Manager
Executors
SparkContext/SparkSession
DAG Scheduler
Task Scheduler
Driver -> The Spark Driver runs inside the ApplicationMaster container (Spark on YARN) in cluster mode, and that container can be launched on any worker node (NodeManager). It does not run on the YARN ResourceManager (master).In Spark on YARN (Cluster Mode), the Driver runs inside the ApplicationMaster container. YARN can launch this container on any worker node (NodeManager). The Driver then requests executor containers from YARN and coordinates the execution of the Spark application.
The Driver is the application manager for a Spark application. In YARN Cluster Mode, it runs inside the ApplicationMaster container on one of the worker nodes (NodeManagers). In Standalone Cluster Mode, the Driver runs as a separate process on one of the cluster nodes (typically a worker). In Client Mode, the Driver runs on the machine that submits the application.
DAG -> Directed acyclic graph
YARN/K8s -> help in resource allocation and maintaining the no of nodes
12. What is SparkContext?
SparkContext is the entry point to Spark Core. It establishes communication with the cluster manager, allocates resources, and creates RDDs.
13.What is SparkSession Explained ?
SparkSession is the unified entry point introduced in Spark 2.x that combines SQLContext, HiveContext, and SparkContext functionality.
14. SparkContext vs SparkSession
15. Repartition() vs Coalesce() ?
Repartition -
Spark = SparkSession.builder.appName(“repartition_app”)
Df = spark.read.parquet(“file.parquet”)
df.repartition(200)
df.coalesce(20)
Repartition
Increase or decrease partitions
Full shuffle
Even distribution
Expensive
Coalesce
Usually decreases partitions
No full shuffle
Faster
Keeps existing partition layout
16. Difference between Cache and Persist ?
Cache is not for "small data spill."
Caching is used to reuse data across multiple actions or computations.
Data spilling is unrelated to cache(). Spilling occurs automatically when Spark runs out of memory.
cache() is actually a shortcut for persist(StorageLevel.MEMORY_AND_DISK) in Spark SQL/DataFrames.
For RDDs, cache() uses MEMORY_ONLY.
For DataFrames/Datasets (Spark 3.x), cache() uses MEMORY_AND_DISK.
17. What is Unpersist ?
unpersist() does not remove the persist configuration. It removes the cached/persisted data from executor memory and/or disk, allowing those resources to be reclaimed.
df.cache()
df.count() # Materializes the cache
df.unpersist() # Removes cached blocks
Important: cache() and persist() are lazy. The data is actually cached only after the first action (such as count(), show(), or collect()).
18.What is the difference between the broadcast variable and Accumulator variable?
Broadcast Variables distribute a read-only copy of a small dataset or object from the Driver to every Executor, reducing communication overhead during tasks such as lookup operations or broadcast joins.
Accumulators are shared variables used for aggregation. Executors can increment or add values, while only the Driver can read the final accumulated result. They are commonly used for counters, metrics, and monitoring job execution.
19. What is shuffling in spark ?
When data is being transferred across nodes for spark execution then it is called shuffle this is slow and expensive (heavy read and write)
why-> if require to do groupBy or repartitionBy(“department”,”city”, 200) or need to join
20. difference between GroupByKey() vs reduceByKey() vs aggregateByKey() vs sortBy() vs sortByKey()
groupByKey() -> It groups all values having the same key.
rdd = sc.parallelize([
("A",1),
("A",2),
("B",3)
])
rdd.groupByKey()
Output:
A -> [1,2]
B -> [3]
reduceByKey() -> Spark performs a map-side (local) combine before shuffling, reducing the amount of data transferred.
aggregateByKey() -> this will do aggregating they keys
sortBy() -> sorting with respect to provided column/attribute
sortByKey() -> sort wrt attribute in a column