Apache Spark
65 questionsPySpark internals, optimization, debugging, Spark SQL, streaming, and real-world scenarios. All examples in PySpark.
RDD vs DataFrame vs Dataset
What's the difference between RDD, DataFrame, and Dataset in Spark? Which should you use and when?
Spark Execution Model
Explain Driver, Executor, Task, Stage, and Job in Spark. What happens when you call df.count()?
Lazy Evaluation
What is lazy evaluation in Spark? What's the difference between transformations and actions? Give examples.
Narrow vs Wide Transformations
What's the difference between narrow and wide transformations? Why does it matter for performance?
Partitions and Parallelism
Your Spark job has 1000 partitions but only 100 executor cores. What happens? How do you choose the right number of partitions?
Catalyst Optimizer
Explain how Catalyst optimizer works. What are the 4 phases of query optimization in Spark?
Shuffle Internals
Explain the Spark shuffle process in detail. What's the difference between hash shuffle and sort shuffle? What's shuffle spill?
Memory Management
Explain Spark's unified memory model. What are execution memory and storage memory? What happens when one runs out?
DAG and Lineage
What is a DAG in Spark? What is lineage? How does Spark use lineage for fault tolerance?
Broadcast Variables vs Accumulators
What are broadcast variables and accumulators? When would you use each in PySpark?
Persist vs Cache vs Checkpoint
Explain cache(), persist(), and checkpoint(). When would you use each?
Spark Session vs Spark Context
What's the difference between SparkSession and SparkContext? Which do you use in modern PySpark?
Broadcast Join
Your join is slow because both tables are large. One is 500MB. How do you optimize it? What's the broadcast threshold?
Handling Data Skew
Your Spark job has one task taking 2 hours while others finish in 5 minutes. Diagnose and fix the skew.
Predicate Pushdown
What is predicate pushdown? How do you verify it's working? When does it fail?
Repartition vs Coalesce
What's the difference between repartition() and coalesce()? When do you use each?
Small File Problem
Your Spark job writes 10,000 tiny Parquet files. Why is this bad? How do you fix it?
UDF Performance
Your PySpark job is slow because of a Python UDF. Why are UDFs slow? What are the alternatives?
Join Strategies Deep Dive
Explain Sort Merge Join vs Shuffle Hash Join vs Broadcast Hash Join. How does Spark choose between them?
Bucketing
What is bucketing in Spark? How does it eliminate shuffles? When should you use it?
AQE (Adaptive Query Execution)
What is AQE in Spark 3? What problems does it solve? Should you always enable it?
Dynamic Partition Pruning
What is Dynamic Partition Pruning (DPP) in Spark? How is it different from static partition pruning?
Window Functions in PySpark
Write PySpark code to: rank employees by salary within each department, calculate running total of sales by date, and get the previous month's revenue using LAG.
Column Operations and Functions
Show how to: add a column, rename a column, drop a column, cast types, handle NULLs, and work with dates in PySpark.
GroupBy and Aggregations
Show how to do groupBy with multiple aggregations, conditional aggregation, and pivot in PySpark.
Reading and Writing Data
Show how to read/write Parquet, CSV, JSON, and Delta in PySpark. How do you handle schema, partitioning, and write modes?
Schema Evolution and Merge
Your daily data feed adds new columns over time. How do you handle schema evolution in PySpark? What's mergeSchema?
Reading Spark UI
Your Spark job is slow. Walk me through how you'd use the Spark UI to diagnose the issue.
OOM Debugging
Your Spark job fails with OutOfMemoryError. How do you determine if it's a driver OOM or executor OOM? How do you fix each?
Job Stuck at Last Few Tasks
Your Spark job has 200 tasks. 198 finish in 2 minutes, the last 2 take 45 minutes. What's happening?
Explain Plan
Read this Spark explain plan and identify the optimization opportunities: df.join(df2, 'id').filter(df.status == 'active').groupBy('city').count().explain()
Structured Streaming Basics
Write a PySpark Structured Streaming job that reads from Kafka, processes JSON events, and writes to a Parquet sink.
Watermarks and Late Data
Your streaming job does windowed aggregation. Events arrive up to 10 minutes late. How do you handle late data with watermarks?
Exactly-Once in Streaming
How does Structured Streaming achieve exactly-once processing? What are the requirements?
Delta Lake Basics
What is Delta Lake? How is it different from plain Parquet? What problems does it solve?
Dedup Streaming Data
Your Kafka topic has duplicate events (at-least-once producer). How do you deduplicate in PySpark — both batch and streaming?
ETL Pipeline Design
Design a PySpark ETL pipeline that reads daily data, validates quality, applies business rules, deduplicates, and writes to a partitioned Delta table. Show the code structure.
Spark Submit and Deploy Modes
Explain spark-submit. What's the difference between client and cluster deploy mode? How do you set executor/driver resources?
Handling Complex/Nested Data
Your JSON data has nested arrays and structs. Show how to flatten, explode, and access nested fields in PySpark.
Spark on EMR vs Databricks vs Glue
Compare running Spark on EMR, Databricks, and AWS Glue. When would you choose each?
Testing Spark Jobs
How do you unit test PySpark code? Show a testing pattern for a transformation function.
Spark vs Pandas
When should you use PySpark vs Pandas? Can you use both together?
Partitioning Data on Write
You're writing a daily ETL output. How do you choose partition columns? What's the impact of too many partition columns?
Spark Configuration Tuning
You have a 50-node cluster with 16 cores and 64GB each. How do you configure spark-submit for optimal resource utilization?
SCD Type 2 in PySpark
Implement SCD Type 2 merge in PySpark using Delta Lake. Handle new records, updated records, and unchanged records.
collect() vs take() vs show()
What's the difference between collect(), take(n), show(n), first(), and head(n)? Which is safe for large datasets?
Joins with Different Column Names
Show how to join DataFrames when: same column name, different column names, multiple columns, and handle duplicate columns after join.
Spark Memory Errors Cheatsheet
Match each error to its cause and fix: 1. java.lang.OutOfMemoryError: Java heap space 2. Container killed by YARN for exceeding memory 3. org.apache.spark.SparkException: Job aborted due to stage failure 4. Python worker exited unexpectedly
Spark vs Flink vs Presto
When would you choose Spark vs Flink vs Presto/Trino for a data engineering task?
Data Lake Architecture with Spark
Design a data lake architecture using Spark. Explain the medallion architecture (Bronze/Silver/Gold) and how each layer is processed.
PySpark Serialization Gotcha
This PySpark code fails with 'PicklingError: Could not serialize object'. What's wrong and how do you fix it? ```python class DataProcessor: def __init__(self, spark): self.spark = spark self.conn = psycopg2.connect('...') def process(self, df): threshold = self.get_threshold() return df.filter(df.value > threshold) def get_threshold(self): cur = self.conn.cursor() cur.execute('SELECT threshold FROM config') return cur.fetchone()[0] ```
Production Incident: Suddenly Slow Job
Your daily Spark ETL job normally takes 30 minutes. Today it's been running for 4 hours. Nothing changed in your code. Walk through your debugging process step by step.
Spark on Kubernetes vs YARN
Your company is migrating from YARN to Kubernetes for Spark. What are the key differences? What breaks? What gets better?
Z-Ordering and Data Skipping
What is Z-ordering? How does it work with data skipping in Delta Lake? When should you Z-order vs partition?
Debugging Slow Joins
Your PySpark join takes 3 hours. Table A is 500GB, Table B is 200GB. Both joined on `user_id`. The cluster has 200 cores and 800GB total memory. Diagnose and optimize.
Cost Optimization for Spark Jobs
Your Spark jobs cost $50K/month on EMR. Management wants you to cut costs by 40% without sacrificing SLAs. What do you do?
Handling Billions of Records
You need to join a 5TB fact table with a 200GB dimension table, deduplicate, aggregate by hour, and write to Delta. The pipeline must complete in under 2 hours. Design the approach.
mapPartitions vs map
When should you use mapPartitions instead of map? Show a real-world example where mapPartitions is significantly faster.
Incremental Processing Patterns
Your daily ETL rewrites the entire table every day (3 hours). How do you convert it to incremental processing to reduce runtime to 15 minutes?
Spark Executor Sizing Anti-patterns
Your team uses these 3 configs. What's wrong with each? 1. `--num-executors 1 --executor-cores 100 --executor-memory 500g` 2. `--num-executors 500 --executor-cores 1 --executor-memory 2g` 3. `--num-executors 50 --executor-cores 5 --executor-memory 4g` (processing 2TB data)
Spark 3.x Features You Should Know
What are the top 5 features in Spark 3.x that changed how you write and tune Spark jobs?
Data Quality Framework in Spark
Design a data quality framework in PySpark. What checks do you run? How do you handle failures? Show code for common quality rules.
foreachBatch Pattern in Streaming
What is foreachBatch in Structured Streaming? When do you use it over regular sinks? Show a real-world example.
Spark SQL vs DataFrame API vs RDD
Write the same transformation in Spark SQL, DataFrame API, and RDD. Which performs best? When would you choose each?
Schema Enforcement vs Evolution
You run a Delta Lake pipeline. Monday, the upstream team adds a new column without telling you. Wednesday, they change a column type. How does Delta handle each case?