Apache Spark Explained — Big Data Processing 2026

Apache Spark is an open-source engine for processing data across a cluster of machines. You write what looks like an ordinary DataFrame or SQL program; Spark turns it into a plan, splits the data into partitions, and runs the pieces in parallel on many computers. When one of those computers fails, Spark rebuilds the lost pieces instead of starting over.

That last property is the idea Spark was built on, and it explains most of how the system behaves, including the parts that confuse newcomers: why nothing happens until you call .show(), why a "shuffle" is expensive, and why .collect() can crash your program. This article walks through where Spark came from, how its execution model works, what the main APIs look like, how Spark has changed in the 4.x releases, and when a single-machine tool such as DuckDB or Polars is the better choice.

---

A Short History: From a Berkeley Lab to Apache

Spark began in 2009 at UC Berkeley's AMPLab, a research lab focused on big data analytics, according to the Apache Software Foundation's 2014 announcement. Its first paper, "Spark: Cluster Computing with Working Sets", appeared at the HotCloud workshop in June 2010. The paper that defined its core abstraction, "Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing" by Matei Zaharia and eight Berkeley colleagues, was presented at USENIX NSDI in April 2012 and won the Best Paper award.

The project entered the Apache Incubator in June 2013 and graduated to a top-level Apache project on 27 February 2014. The ASF said that during incubation it had drawn code from more than 120 developers at 25 organizations, and listed Alibaba, Cloudera, Databricks, IBM, Intel and Yahoo among its users at the time.

The research that followed added most of the components people use today. The project's own research page lists them:

Year	Paper	Venue	What it added
2010	Spark: Cluster Computing with Working Sets	HotCloud	The original system
2012	Resilient Distributed Datasets	NSDI (Best Paper)	The RDD abstraction and fault model
2013	Shark: SQL and Rich Analytics at Scale	SIGMOD	SQL on Spark, forerunner of Spark SQL
2013	Discretized Streams	SOSP	Streaming as a series of small batches
2014	GraphX	OSDI	Graph processing on Spark
2015	Spark SQL: Relational Data Processing in Spark	SIGMOD	DataFrames and the Catalyst optimizer
2016	MLlib: Machine Learning in Apache Spark	JMLR vol. 17	The distributed ML library
2016	SparkR	SIGMOD	R bindings

The problem Spark was solving

Before Spark, the standard way to process data across a cluster was Hadoop MapReduce. MapReduce runs a job as a map step and a reduce step, and writes the results to replicated distributed storage between steps. That is safe, but the Spark research page describes the cost plainly: such engines are "inefficient for multi-pass applications" because they pay to load data on every step and write it back to replicated storage.

Many important workloads are multi-pass. Iterative machine learning algorithms touch the same data dozens of times; an analyst exploring a dataset runs query after query against it. The NSDI paper argued that keeping the working set in memory could speed up these workloads "by an order of magnitude", and the project's research page states that RDDs let Spark "outperform existing models by up to 100x in multi-pass analytics". Treat "up to 100x" as a best case for iterative, memory-resident workloads, not a general speedup; on a single pass over data that does not fit in memory, the gap is much smaller.

---

The Core Idea: Resilient Distributed Datasets

A resilient distributed dataset (RDD) is a read-only collection of records split into partitions spread across the cluster. Three properties define it.

1. It is immutable. You never edit an RDD. A transformation such as map or filter produces a new RDD.
2. It is built by coarse-grained transformations. Each RDD records the operation, applied to every record, that produced it from its parent. The NSDI paper calls this a "restricted form of shared memory" based on coarse-grained transformations rather than fine-grained updates to individual values.
3. It knows its lineage. Because every RDD remembers how it was derived, Spark can rebuild a lost partition by re-running the transformations on the parent data. It does not need to keep replicated copies of intermediate results.

Lineage is the "resilient" part. A system that lets you update any value anywhere must log every update or replicate the data to survive a failure, which is expensive. A system that only allows whole-dataset transformations can log a few lines of recipe instead. The trade-off is expressiveness: RDDs are a poor fit for workloads built on many small random updates, such as an online transaction database. The paper argued they were still expressive enough to capture specialized models such as Google's Pregel graph system.

Most people no longer write RDD code directly. Since Spark SQL arrived, the DataFrame (a distributed table with named columns and a schema) has been the main API, and in the Scala and Java APIs a typed variant called the Dataset. DataFrames still run on the same execution engine, but they give Spark's optimizer far more information about what you are doing, which is why they are usually faster than hand-written RDD code.

---

How a Spark Program Actually Runs

The cluster overview in the official documentation describes the moving parts. A Spark application runs as a set of independent processes on a cluster, coordinated by one central process.

Component	What it is (per the Spark docs)
Driver program	The process running your main() function and creating the SparkContext (wrapped by SparkSession in modern code)
Cluster manager	An external service that allocates resources: Spark's own Standalone manager, Hadoop YARN, or Kubernetes
Executor	A process launched for your application on a worker node; it runs tasks and keeps data in memory or on disk. Each application gets its own executors
Job	A parallel computation spawned in response to an action such as save or collect
Stage	A set of tasks within a job; stages depend on one another, much like map and reduce phases
Task	A unit of work sent to one executor, usually one partition of data

The documentation adds a practical warning: the driver must be reachable over the network from the workers and should run close to them, ideally on the same local network, because it schedules every task.

Lazy evaluation and the DAG

Spark separates transformations from actions. A transformation (filter, select, groupBy, join) describes a new dataset and returns immediately; nothing is computed. An action (count, show, collect, write) asks for a result, and only then does Spark build and run a plan.

from pyspark.sql import SparkSession, functions as F

spark = (SparkSession.builder
         .appName("sales-report")
         .getOrCreate())

df = spark.read.parquet("s3://bucket/sales/")

# Transformations: these only build a plan
result = (df.filter(F.col("amount") > 100)
            .groupBy("region")
            .agg(F.sum("amount").alias("total"))
            .orderBy(F.col("total").desc()))

# Action: now Spark plans, schedules, and runs the work
result.show()

Laziness is not a quirk; it is what lets Spark optimize. Because it sees the whole chain before running anything, it can push the filter down into the Parquet reader so that rows with small amounts are never loaded, read only the two columns the query uses, and combine adjacent steps into one pass. You can inspect the result with result.explain(), which prints the physical plan Spark intends to run.

Narrow and wide transformations

The plan is a directed acyclic graph (DAG) of operations, and Spark cuts it into stages at one specific kind of boundary.

• A narrow transformation, such as filter or select, needs only the data already in one partition. Spark can chain many of them inside a single task.
• A wide transformation, such as groupBy, join or orderBy, needs records from many partitions to meet. Spark must shuffle: each task writes its output split by key, and the next stage's tasks fetch the pieces they need over the network.

Shuffles are where most Spark jobs spend their time and where most performance problems live. They involve disk writes, network transfer, and serialization, and they are where data skew bites: if one key (say, one very large customer) holds far more rows than the rest, the task that receives it runs much longer than everyone else, and the whole stage waits for it. In the Spark UI, that pattern shows up as a stage where almost every task finishes in seconds and one runs for many minutes.

---

The Main APIs

DataFrames and PySpark

PySpark is the Python API, and the DataFrame is the object most code works with. If you know pandas, the concepts transfer; the execution model does not. The DuckDB vs Pandas comparison covers the single-machine side of the same trade-offs.

# pandas: eager, single machine
import pandas as pd
pdf = pd.read_csv("data.csv")
out = pdf[pdf["amount"] > 100].groupby("region")["amount"].sum()

# PySpark: lazy, distributed
sdf = spark.read.csv("data.csv", header=True, inferSchema=True)
out = sdf.filter(F.col("amount") > 100).groupBy("region").agg(F.sum("amount"))

	pandas	PySpark DataFrame
Execution	Eager: each line runs immediately	Lazy: runs when an action is called
Where data lives	One process's memory	Partitions across executors
Row order and index	Has an index; order is meaningful	No index; order only after an explicit sort
Random row access	Cheap (.iloc)	Not a natural operation
Fault tolerance	None; a crash loses the work	Lost partitions recomputed from lineage

The "no index" row matters more than it looks. Code that relies on row position, such as "take the previous row" without an explicit ordering column, has no stable meaning in Spark, because partitions can be processed in any order. Use window functions with an explicit orderBy instead.

Spark SQL

Every DataFrame can be queried with SQL, and both go through the same optimizer, so the choice is about readability rather than speed.

df.createOrReplaceTempView("sales")

result = spark.sql("""
    SELECT region, SUM(amount) AS total
    FROM sales
    WHERE amount > 100
    GROUP BY region
    ORDER BY total DESC
""")

The 2015 SIGMOD paper that introduced Spark SQL also introduced Catalyst, the query optimizer that rewrites logical plans (predicate pushdown, column pruning, constant folding) before choosing physical operators. Teams that already think in SQL often write most transformations this way, and tools such as dbt can generate the SQL; our dbt tutorial shows that workflow on a warehouse.

MLlib

MLlib is Spark's distributed machine learning library, described in a 2016 paper in the Journal of Machine Learning Research. It provides feature transformers, classical algorithms (linear models, decision trees and forests, gradient-boosted trees, clustering, collaborative filtering) and a pipeline API modeled on scikit-learn's.

from pyspark.ml.feature import VectorAssembler
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.evaluation import BinaryClassificationEvaluator

assembler = VectorAssembler(inputCols=["age", "income", "spending"],
                            outputCol="features")
data = assembler.transform(df)
train, test = data.randomSplit([0.8, 0.2], seed=42)

model = RandomForestClassifier(featuresCol="features",
                               labelCol="target",
                               numTrees=100).fit(train)

auc = BinaryClassificationEvaluator(labelCol="target").evaluate(model.transform(test))

MLlib's catalogue is smaller than scikit-learn's, and it is not a deep learning framework. Its practical niche is feature engineering and classical models on data too large to sample down. If the training set fits on one machine after aggregation, single-node libraries are usually simpler; see What Is Machine Learning? for the underlying ideas, and Feature Stores Explained for how Spark-computed features are often served to models.

Structured Streaming

Structured Streaming lets you write a streaming job with the same DataFrame API. Conceptually, a stream is an unbounded table that keeps growing, and your query is re-run incrementally on new rows. Its ancestor was the 2013 Discretized Streams work, which treated a stream as a sequence of small batch jobs.

stream = (spark.readStream.format("kafka")
          .option("kafka.bootstrap.servers", "kafka:9092")
          .option("subscribe", "events")
          .load())

counts = (stream.selectExpr("CAST(value AS STRING) AS json")
          .select(F.from_json("json", schema).alias("e")).select("e.*")
          .withWatermark("event_time", "10 minutes")
          .groupBy(F.window("event_time", "5 minutes"), "user_id")
          .count())

query = counts.writeStream.outputMode("update").format("console").start()

The watermark tells Spark how late an event may arrive and still be counted; it lets Spark drop old aggregation state instead of keeping it forever. The appeal of Structured Streaming is one codebase for batch and streaming. Systems designed purely for streaming, such as Apache Flink, are a common alternative when per-event latency is the priority. For forecasting on the resulting time series, see How Time Series Forecasting Works.

---

What Changed in Spark 4

The Apache Spark 4.0.0 release notes describe the first 4.x release, published on 23 May 2025, as resolving more than 5,100 JIRA tickets with contributions from more than 390 people. Changes worth knowing:

• ANSI SQL mode is on by default. Operations that used to return NULL silently, such as an invalid cast or an integer overflow, now raise errors. Old jobs that relied on the lenient behaviour can fail after upgrading; that is the most common migration surprise.
• Spark Connect matured. Spark Connect separates a thin client from the cluster, so a notebook or application talks to Spark over a network protocol instead of running a full driver locally. Spark 4.0 shipped a lightweight Python client (pyspark-client, about 1.5 MB), a spark.api.mode setting, and ML support over Connect.
• A VARIANT data type for semi-structured data such as JSON.
• A Python Data Source API, so connectors can be written in Python, plus Python user-defined table functions.
• SQL user-defined functions, session variables, SQL pipe syntax, string collation, and a built-in XML data source.
• Platform floors moved: Scala 2.13 and JDK 17 became the defaults, with Scala 2.12, JDK 8 and JDK 11 dropped, and Python 3.8 support removed.

Releases have continued since. According to the project news page, Spark 4.2.0 was released on 14 July 2026, alongside maintenance releases 4.1.3, 4.0.4 and 3.5.9 that month. Check that page before pinning a version, because several release lines are maintained at once.

---

Performance: What the Documentation Recommends

Let Adaptive Query Execution do its job

The SQL performance tuning guide describes Adaptive Query Execution (AQE), which re-optimizes a query while it runs using statistics gathered at shuffle boundaries. It has been enabled by default since Spark 3.2.0 and does three main things:

AQE feature	Setting (default)	What it does
Coalescing post-shuffle partitions	spark.sql.adaptive.coalescePartitions.enabled (true)	Merges small shuffle partitions toward a target size (advisoryPartitionSizeInBytes, 64 MB)
Skew join handling	spark.sql.adaptive.skewJoin.enabled (true)	Splits oversized partitions in sort-merge joins into evenly sized tasks
Join strategy switching	spark.sql.adaptive.autoBroadcastJoinThreshold	Turns a sort-merge join into a broadcast join when one side turns out to be small at runtime

The static setting spark.sql.shuffle.partitions still defaults to 200, and spark.sql.autoBroadcastJoinThreshold defaults to 10 MB. Before AQE, tuning the first by hand was one of the main jobs of a Spark engineer; with AQE, the guide notes, coalescing removes much of the need to pick that number precisely.

Store data in columnar formats

Parquet and ORC files store each column separately, with statistics per block. That lets Spark read only the columns a query needs and skip blocks whose min/max values cannot match a filter. CSV offers neither, and reading it with inferSchema=True costs an extra pass over the data to guess types. Table formats such as Delta Lake, Apache Iceberg and Apache Hudi build on Parquet and add transactions and schema evolution.

# Partitioned output: later reads can skip whole directories
df.write.partitionBy("year", "month").parquet("s3://bucket/sales_by_month/")

Partition by columns you filter on and that have modest cardinality. Partitioning by a column with millions of values (such as user_id) produces millions of tiny files, which is its own performance problem.

Cache only what you reuse

df.cache() (or spark.catalog.cacheTable("name")) keeps a dataset in executor memory after the first action computes it. It helps when the same intermediate result feeds several later actions, as in an iterative algorithm. It hurts when you cache something used once, because it takes memory away from execution. Release it with df.unpersist().

Keep results on the cluster

rows = df.collect()            # pulls every row into the driver; can exhaust its memory
local = df.limit(1000).toPandas()   # bounded sample is fine
df.write.parquet("s3://bucket/output/")   # large results stay distributed

Prefer built-in functions to Python UDFs

A Python user-defined function forces data to move between the JVM executors and Python worker processes and hides the logic from the optimizer. Built-in functions in pyspark.sql.functions run inside the engine. When custom Python logic is unavoidable, vectorized pandas UDFs, which process batches through Apache Arrow, are generally much cheaper than row-at-a-time UDFs.

---

Where Spark Runs

Spark itself is free, open-source software under the Apache License 2.0. You can run it in local mode on a laptop (useful for development and tests), on its built-in Standalone cluster manager, on Hadoop YARN, or on Kubernetes, which are the three cluster managers the current documentation lists.

Most production deployments use a managed service, which handles provisioning, autoscaling and upgrades. The common ones are Databricks (founded by members of the original Berkeley team), Amazon EMR, Google Cloud Dataproc, and Spark pools in Microsoft's Azure analytics services. The managed platforms add their own runtimes and proprietary optimizations on top of open-source Spark, so behaviour and performance can differ between them for the same code.

---

Spark vs Single-Node Tools and Warehouses

The biggest change in the data landscape since Spark's early years is not a rival cluster engine. It is that single machines got large and single-node engines got fast. A cloud server with hundreds of gigabytes of RAM and fast local storage can handle datasets that once required a cluster, and engines such as DuckDB and Polars use columnar, vectorized, multi-threaded execution to do it. Our explainers on DuckDB and Polars vs Pandas go into how.

	Apache Spark	DuckDB / Polars	Cloud warehouse (Snowflake, BigQuery)
Scale-out across machines	Yes, by design	No; one machine, many cores	Yes, managed for you
Setup	Cluster or managed service	A library install	An account; no servers to run
Startup overhead	Seconds (JVM, executors)	Milliseconds	Low for queries
Main interface	Python, Scala, Java, R, SQL	SQL and Python (DuckDB); Python and Rust (Polars)	SQL
Fault tolerance mid-job	Lineage-based recomputation	Restart the job	Managed by the service
Typical strength	Large multi-stage ETL, streaming plus batch, ML feature pipelines	Interactive analysis and pipelines that fit on one machine	Shared analytics on governed tables

There is no clean size threshold where Spark becomes necessary; it depends on memory, query shape, and how often the job runs. A useful rule is to ask whether the work genuinely needs more than one machine, either because the data is too big for one or because the job must survive machine failure partway through. If not, the single-node tools are simpler and often faster, because they skip network shuffles and JVM startup. For warehouse-centric teams, see Snowflake vs BigQuery vs Redshift; for how the pieces fit in a typical stack, see Best Data Science Tools in 2026.

When Spark is still the natural choice

• Data and intermediate results that exceed one machine, especially in joins between two very large tables, where the shuffle itself is the work.
• Long multi-stage pipelines where restarting from scratch after a single node failure would be costly; lineage recovery is exactly what Spark was designed for.
• One codebase for batch and streaming, using Structured Streaming.
• An existing lakehouse on Delta Lake or Iceberg with Spark-based tooling around it, where migration costs are real.
• Feature engineering for ML over full datasets, where sampling is not acceptable.

---

Common Misconceptions

"Spark is an in-memory database." It is a processing engine, not a storage system. It reads from and writes to external storage (object stores, HDFS, databases). "In-memory" refers to keeping working data in executor memory during a computation, and Spark spills to disk when memory runs out.

"Spark is always faster than a single machine." The coordination costs of a cluster (scheduling, serialization, network shuffles) are real. For data that fits on one machine, a well-built single-node engine frequently finishes first.

"Caching everything speeds things up." Cache competes with execution for executor memory. Cache only datasets reused across several actions.

"More partitions means more parallelism, so more is better." Each task has scheduling overhead, and many tiny partitions produce many tiny output files. Too few partitions waste cores and risk out-of-memory errors. AQE's coalescing exists because no single static number is right for every stage.

"PySpark is slow because Python is slow." For DataFrame and SQL code, the Python program only builds the plan; execution happens inside the JVM engine. Python speed matters mainly when you use row-at-a-time Python UDFs.

Getting the data clean before it reaches Spark matters as much as any tuning; What Data Cleaning Actually Involves covers the work that usually dominates a pipeline's effort.

---

Learning Path

1. Run Spark locally. pip install pyspark and use SparkSession.builder.master("local[*]") to learn the API on your laptop.
2. Learn the DataFrame API and Spark SQL, including window functions and joins. The Python Data Libraries Cheat Sheet is a handy companion for the pandas-side equivalents.
3. Read the Spark UI. Learn to find stages, shuffle read and write sizes, and skewed tasks. Most real tuning starts here.
4. Read explain() output to see filters pushed down and join strategies chosen.
5. Move to a cluster or managed service and learn the deployment model you will use in production.

---

FAQ

Q: Is Apache Spark free? A: Yes. Spark is open-source software under the Apache License 2.0 and can be run on your own machines at no licence cost. Managed services such as Databricks, Amazon EMR and Google Cloud Dataproc charge for the infrastructure and platform features they add.

Q: What does "resilient" mean in resilient distributed dataset? A: It means a lost partition can be rebuilt. Each RDD records the coarse-grained transformations that produced it from its parent data, so after a machine failure Spark re-runs those steps for the missing partitions instead of relying on replicated copies, as described in the 2012 NSDI paper by Zaharia and colleagues.

Q: Why does nothing happen until I call show() or count()? A: Spark evaluates lazily. Transformations such as filter and groupBy only build a plan; an action triggers a job. Waiting lets the optimizer see the whole query, push filters into the file reader, and skip unneeded columns before any data moves.

Q: Is Spark the same as Databricks? A: No. Spark is the open-source Apache project. Databricks is a commercial platform, founded by members of Spark's original Berkeley team, that runs its own Spark-based runtime with additional proprietary features and management tools.

Q: Should I use PySpark or Scala? A: For DataFrame and SQL work, both build the same plan for the same engine, so PySpark is the common choice for data teams. Scala or Java make more sense when you are writing Spark extensions, libraries, or heavy custom logic that would otherwise need Python UDFs.

Q: When should I not use Spark? A: When the data and its intermediate results fit comfortably on one machine and the job does not need to survive node failures. Single-node engines such as DuckDB and Polars avoid cluster overhead and are simpler to operate; cloud warehouses suit SQL analytics on shared tables.

Q: What is the biggest change in Spark 4? A: For most existing jobs, ANSI SQL mode becoming the default in Spark 4.0. Invalid casts and arithmetic overflows that once returned NULL now raise errors, so test jobs before upgrading. Spark Connect's thin clients are the largest architectural change.

---

Sources

• Zaharia, M. et al. (2012). Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing. USENIX NSDI '12. usenix.org
• Apache Spark — Research papers list. spark.apache.org/research.html
• Apache Software Foundation (27 Feb 2014). The ASF Announces Apache Spark as a Top-Level Project. news.apache.org
• Apache Spark documentation — Cluster Mode Overview. spark.apache.org
• Apache Spark documentation — Performance Tuning (Spark SQL). spark.apache.org
• Apache Spark 4.0.0 release notes. spark.apache.org
• Apache Spark news and release history. spark.apache.org/news

---

Related Articles

• DuckDB vs Pandas
• Polars vs Pandas
• Snowflake vs BigQuery vs Redshift
• dbt Tutorial

Last verified: October 2026