Apache Spark Fundamentals — Big Data Processing at Scale

When your data outgrows a single machine, you need Apache Spark. A pandas DataFrame on your laptop handles millions of rows. Spark handles billions to trillions — by distributing the work across hundreds or thousands of machines.

Spark is the dominant engine for large-scale data processing. It powers ETL pipelines, machine learning workflows, streaming applications, and analytics at companies processing petabytes of data daily. Understanding Spark is essential for any data engineer working beyond the limits of a single database.

Why Spark replaced MapReduce

Before Spark, Hadoop MapReduce was the standard for distributed data processing. MapReduce worked, but it was painful:

MapReduce wrote to disk after every step. A multi-step job (read, filter, join, aggregate) wrote intermediate results to HDFS between each Map and Reduce phase. This made complex pipelines extremely slow.

Spark keeps data in memory. Instead of writing to disk after every operation, Spark chains operations together and keeps intermediate results in RAM. For iterative algorithms (like machine learning), this is 10-100x faster.

MapReduce required Java boilerplate. Even simple operations required dozens of lines of Java code — Mapper classes, Reducer classes, driver programs, serialization. Spark’s API is concise: the same logic takes 3-5 lines in PySpark.

# MapReduce word count: ~50 lines of Java
# Spark word count: 3 lines of Python
counts = (
    spark.read.text("input.txt")
    .select(explode(split(col("value"), " ")).alias("word"))
    .groupBy("word").count()
)

Spark did not just improve performance — it made distributed computing accessible to data engineers who think in SQL and Python, not Java and JVM internals.

Core abstractions

Spark has evolved through three abstraction layers. Understanding all three helps you read legacy code and choose the right API.

RDDs (Resilient Distributed Datasets)

The original Spark API. An RDD is an immutable, distributed collection of objects. You create them by parallelizing existing data or reading from storage, then chain transformations.

# RDD API — low-level, rarely used directly today
rdd = sc.parallelize([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
result = (
    rdd
    .filter(lambda x: x % 2 == 0)     # Keep even numbers
    .map(lambda x: x ** 2)            # Square them
    .reduce(lambda a, b: a + b)       # Sum
)
# Result: 4 + 16 + 36 + 64 + 100 = 220

RDDs are untyped — Spark does not know the schema of your data, so it cannot optimize the execution plan. Think of RDDs as a bag of Python objects that happen to be distributed.

When to use: Almost never in modern Spark. Needed only for unstructured data that does not fit into rows and columns, or for very specialized low-level operations.

DataFrames

The modern Spark API. A DataFrame is a distributed collection of rows with named columns — like a database table or a pandas DataFrame, but distributed across a cluster.

# DataFrame API — the standard for modern Spark
from pyspark.sql import SparkSession
from pyspark.sql.functions import col, sum, avg, count, when

spark = SparkSession.builder.appName("example").getOrCreate()

# Read data
orders = spark.read.parquet("s3://data-lake/orders/")

# Transform
result = (
    orders
    .filter(col("status") == "completed")
    .groupBy("customer_id", "region")
    .agg(
        count("order_id").alias("total_orders"),
        sum("amount").alias("total_revenue"),
        avg("amount").alias("avg_order_value")
    )
    .filter(col("total_orders") >= 5)
    .orderBy(col("total_revenue").desc())
)

DataFrames are schema-aware. Spark knows the column names and types, which enables the Catalyst optimizer to create efficient execution plans. This is the API you should use 95% of the time.

SparkSQL

Write SQL against DataFrames. Many data engineers prefer this because the logic reads like standard SQL:

# Register DataFrame as a temporary view
orders.createOrReplaceTempView("orders")

# Write SQL
result = spark.sql("""
    SELECT
        customer_id,
        region,
        COUNT(order_id) AS total_orders,
        SUM(amount) AS total_revenue,
        AVG(amount) AS avg_order_value
    FROM orders
    WHERE status = 'completed'
    GROUP BY customer_id, region
    HAVING COUNT(order_id) >= 5
    ORDER BY total_revenue DESC
""")

SparkSQL and the DataFrame API produce identical execution plans. The Catalyst optimizer compiles both to the same physical plan. Choose whichever your team reads more easily.

The Catalyst optimizer

This is Spark’s secret weapon. When you write a query (DataFrame or SQL), Catalyst:

1. Parses it into a logical plan.
2. Analyzes column references and types.
3. Optimizes by reordering operations (push filters before joins, prune unused columns).
4. Generates physical execution code.

Your query (DataFrame or SQL)
    ↓ Parse
Unresolved logical plan
    ↓ Analyze
Resolved logical plan
    ↓ Optimize (predicate pushdown, column pruning, join reordering)
Optimized logical plan
    ↓ Physical planning
Physical plan (the actual operations Spark will execute)
    ↓ Code generation (Tungsten)
Optimized JVM bytecode

Because of Catalyst, you can write readable code and trust the optimizer to make it fast. This is why DataFrames outperform RDDs — with RDDs, Spark cannot see inside your lambda functions to optimize.

The execution model

Understanding how Spark executes your code is essential for debugging performance issues.

Architecture

┌─────────────────────────────────────────────────────┐
│                    DRIVER                            │
│  - SparkSession / SparkContext                       │
│  - Builds execution plan                             │
│  - Coordinates executors                             │
│  - Collects results                                  │
└────────────┬────────────┬────────────┬──────────────┘
             │            │            │
     ┌───────▼──┐  ┌──────▼───┐  ┌────▼─────┐
     │Executor 1│  │Executor 2│  │Executor 3│
     │          │  │          │  │          │
     │ Task A   │  │ Task C   │  │ Task E   │
     │ Task B   │  │ Task D   │  │ Task F   │
     │          │  │          │  │          │
     │ Cache    │  │ Cache    │  │ Cache    │
     └──────────┘  └──────────┘  └──────────┘

Driver: The process that runs your main program. It creates the SparkSession, builds the execution plan, and coordinates work. The driver does not process data — it orchestrates.

Executors: Worker processes on cluster nodes. Each executor runs tasks (units of work) and stores cached data in memory. An executor might have 4 cores and 16 GB of RAM, running 4 tasks in parallel.

Tasks: The smallest unit of work. Each task processes one partition of data. If your data has 200 partitions, Spark creates 200 tasks and distributes them across executors.

Jobs, stages, and tasks

# This single line of code becomes multiple stages
result = (
    orders                              # Stage 1: Read + filter
    .filter(col("status") == "completed")
    .join(customers, "customer_id")     # Stage 2: Shuffle for join
    .groupBy("region")                  # Stage 3: Shuffle for aggregate
    .agg(sum("amount").alias("revenue"))
    .collect()                          # Action triggers execution
)

A job is triggered by an action (collect, write, count). A job contains multiple stages, separated by shuffles (data redistribution across executors). Each stage contains multiple tasks, one per partition.

The key insight: Shuffles are expensive. They require writing data to disk, sending it across the network, and reading it back. Minimizing shuffles is the primary way to optimize Spark performance.

PySpark basics

Reading data

from pyspark.sql import SparkSession

spark = SparkSession.builder \
    .appName("my_pipeline") \
    .config("spark.sql.adaptive.enabled", "true") \
    .getOrCreate()

# Parquet (columnar, compressed — the default for Spark)
orders = spark.read.parquet("s3://data-lake/orders/")

# CSV
raw_csv = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv("s3://data-lake/uploads/orders.csv")

# JSON
events = spark.read.json("s3://data-lake/events/")

# Delta Lake (versioned, ACID-compliant)
customers = spark.read.format("delta").load("s3://data-lake/customers/")

# JDBC (database)
pg_orders = spark.read \
    .format("jdbc") \
    .option("url", "jdbc:postgresql://host:5432/db") \
    .option("dbtable", "public.orders") \
    .option("user", "reader") \
    .option("password", "secret") \
    .load()

Transformations (lazy)

Transformations define what to do but do not execute until an action is called:

from pyspark.sql.functions import (
    col, lit, when, coalesce, date_format,
    sum, avg, count, max, min,
    year, month, datediff, current_date,
    explode, split, trim, lower, upper,
    row_number, rank, dense_rank, lag, lead
)
from pyspark.sql.window import Window

# Filter
active_orders = orders.filter(
    (col("status") == "completed") &
    (col("amount") > 0) &
    (col("ordered_at") >= "2024-01-01")
)

# Add computed columns
enriched = active_orders.withColumn(
    "order_year", year(col("ordered_at"))
).withColumn(
    "days_since_order", datediff(current_date(), col("ordered_at"))
).withColumn(
    "amount_tier",
    when(col("amount") >= 1000, "high")
    .when(col("amount") >= 100, "medium")
    .otherwise("low")
)

# Window functions
window_spec = Window.partitionBy("customer_id").orderBy("ordered_at")

with_sequence = enriched.withColumn(
    "order_number", row_number().over(window_spec)
).withColumn(
    "prev_order_amount", lag("amount", 1).over(window_spec)
).withColumn(
    "running_total", sum("amount").over(
        window_spec.rowsBetween(Window.unboundedPreceding, Window.currentRow)
    )
)

Actions (trigger execution)

# Collect results to driver (small results only!)
top_customers = result.limit(100).collect()

# Write to storage
result.write \
    .mode("overwrite") \
    .partitionBy("order_year", "region") \
    .parquet("s3://data-lake/processed/orders/")

# Write to Delta Lake
result.write \
    .format("delta") \
    .mode("overwrite") \
    .save("s3://data-lake/delta/orders/")

# Count
print(f"Total rows: {orders.count()}")

# Show sample
orders.show(10, truncate=False)

Spark platforms: Databricks vs EMR vs Dataproc

Databricks

The company founded by the creators of Spark. Databricks provides the most polished Spark experience:

• Optimized Spark runtime (Photon engine, 2-5x faster than open-source Spark)
• Unity Catalog for governance
• Collaborative notebooks
• Delta Lake native
• Built-in MLflow for ML workflows

Best for: Teams that want a fully managed data platform with performance optimization out of the box.

Amazon EMR

AWS’s managed Hadoop/Spark service:

• Tight integration with S3, Glue Catalog, Redshift
• Pay-per-second pricing with Spot instances
• EMR Serverless for zero-infrastructure management
• More configuration control than Databricks

Best for: AWS-native teams comfortable with infrastructure management, cost-sensitive workloads using Spot instances.

Google Dataproc

GCP’s managed Spark service:

• Auto-scaling clusters
• Tight BigQuery integration
• Sub-minute cluster startup
• Preemptible VMs for cost savings

Best for: GCP-native teams, especially those using BigQuery as their primary warehouse.

When to use Spark vs warehouse SQL

This is the question every data engineer faces. The answer depends on your data volume and complexity:

Scenario	Use Spark	Use warehouse SQL (dbt)
Data volume	Terabytes to petabytes	Gigabytes to terabytes
Transformation type	Complex Python logic, ML	SQL transformations
Data format	Files (Parquet, JSON, CSV)	Tables in warehouse
Team skills	Python/Scala engineers	SQL-fluent analysts
Latency requirements	Batch (minutes to hours)	Batch (seconds to minutes)
Unstructured data	Yes (logs, JSON, text)	Limited

The modern trend: Cloud warehouses (Snowflake, BigQuery, Redshift) are getting faster and handling larger datasets. Many transformations that required Spark five years ago can now be done with dbt + warehouse SQL. Use Spark when you need Python/ML capabilities or when data volume truly exceeds warehouse capacity.

Performance optimization

1. Partition your data

Partitioning controls how data is distributed across files and executors:

# Write data partitioned by date — queries on specific dates read fewer files
df.write \
    .partitionBy("year", "month") \
    .parquet("s3://data-lake/events/")

# When reading, Spark only reads relevant partitions
events_jan = spark.read.parquet("s3://data-lake/events/") \
    .filter(col("year") == 2024) \
    .filter(col("month") == 1)
# Only reads files under year=2024/month=1/

2. Cache strategically

# Cache a DataFrame that's used multiple times
customer_orders = orders.join(customers, "customer_id")
customer_orders.cache()  # Stores in executor memory

# Now both of these use the cached version
total_by_region = customer_orders.groupBy("region").agg(sum("amount"))
total_by_segment = customer_orders.groupBy("segment").agg(sum("amount"))

# Unpersist when done
customer_orders.unpersist()

Cache judiciously — memory is limited. Only cache DataFrames that are reused multiple times and are expensive to recompute.

3. Broadcast joins

When joining a large table with a small table, broadcast the small table to avoid a shuffle:

from pyspark.sql.functions import broadcast

# Without broadcast: both tables are shuffled (expensive)
result = big_table.join(small_table, "key")

# With broadcast: small_table is sent to all executors (fast)
result = big_table.join(broadcast(small_table), "key")

Spark auto-broadcasts tables under 10 MB by default (spark.sql.autoBroadcastJoinThreshold). Increase this threshold if your “small” table is 50-100 MB.

4. Avoid common anti-patterns

# BAD: Collecting large results to the driver
all_data = huge_df.collect()  # OutOfMemoryError on driver

# GOOD: Keep processing distributed
huge_df.write.parquet("s3://output/")

# BAD: Using Python UDFs (bypass Catalyst optimizer)
from pyspark.sql.functions import udf
slow_udf = udf(lambda x: x.upper())  # Serializes to Python, slow

# GOOD: Use built-in functions (run in optimized JVM)
from pyspark.sql.functions import upper
fast = df.withColumn("name", upper(col("name")))

# BAD: Too many small partitions (overhead per task)
# GOOD: Repartition to right size
df = df.repartition(200)  # For large datasets
df = df.coalesce(10)       # Reduce partitions without shuffle

5. Enable Adaptive Query Execution (AQE)

spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")

AQE dynamically adjusts the execution plan based on runtime statistics. It coalesces small partitions, handles skewed joins, and switches join strategies — all automatically. Enable it. There is almost no downside.

Next steps

• Data Warehouse Concepts — understand the warehouses Spark feeds data into.
• dbt Fundamentals — when warehouse SQL is enough, dbt is the tool.
• Data Pipeline Testing — test your Spark transformations.
• Pipeline CI/CD — deploy Spark jobs with confidence.