Spark Performance Tuning: Why Is Your Apache Spark Job Slow — and How Do You Fix It?

Overview

Spark performance tuning is mostly a diagnosis problem. The fixes are well known — the hard part is figuring out which of the six usual suspects is hurting this particular job: data skew, shuffle, spill, GC pressure, small files, or a bad query plan. Tuning configs before you’ve identified the cause is how teams end up with 40-line spark-submit incantations that fix nothing.

This page is the hub. Each section below covers one failure mode: the symptom, the mechanism, where to confirm it in the Spark UI, and the fix — with a deep dive linked for each. Start with the table.

Cause	Tell-tale symptom in the Spark UI	Quick fix	Deep dive
Data skew / stragglers	Stages → Summary Metrics: max task duration ≫ median	AQE skew join, salt the hot key	Spark data skew
Shuffle	Huge Shuffle Read/Write on a stage; high fetch wait	Cut wide operations, size partitions	Low-shuffle MERGE
Spill	Spill (Memory) / Spill (Disk) columns appear on a stage	More partitions or more memory per task	Accelerator memory & OOM
GC pressure	Executors tab: GC Time red, >10% of task time	Smaller heaps, more executors, off-heap	What GC overhead costs
Small files	Scan node reads 100k+ files; thousands of sub-second tasks	Compact / cluster the table	Clustering 4× faster
Bad query plan	SortMergeJoin where a broadcast was expected	Join hints, refresh table stats	SHJ vs SMJ

Why is my Spark job stuck on the last task?

199 tasks finish in a minute; task 200 runs for an hour. That’s data skew: partitioning hashed one hot key (a null, a default value, one giant customer) into a single partition, and the whole stage waits on the one task processing it.

Confirm it in the Stages tab → Summary Metrics for completed tasks. Healthy stages show max duration within ~2× of the 75th percentile. Skewed stages show a max that’s 10–100× the median, and the sorted task table shows the same task with an outsized shuffle read size.

The fix, in order of effort: enable AQE skew handling (on by default in Spark 3.2+):

spark.sql.adaptive.enabled=true
spark.sql.adaptive.skewJoin.enabled=true

If AQE can’t split it (skewed aggregations, exploded joins), salt the hot key — the data skew deep dive linked above walks through the pattern.

The Quanton Live Monitor tracks skew ratio per stage at a 2-second refresh, and Diagnostics raises a data-skew alert the moment a straggler emerges — while the job is still running.

Why is shuffle the bottleneck?

The stage is slow but no single task stands out — everything is waiting on the network. Every wide operation (joins, groupBy, repartition, window functions) shuffles data between executors: serialize, write to disk, transfer, fetch, deserialize. When a job moves hundreds of GiB per shuffle, that movement is the runtime.

Confirm it in the Stages tab: Shuffle Read and Shuffle Write columns. If a stage shuffles a large multiple of what it eventually outputs, the plan is moving data it doesn’t need to. High shuffle read fetch wait time in task metrics means executors sit idle waiting on remote blocks.

Fixes: eliminate shuffles before shrinking them — broadcast the small side of joins, pre-aggregate before wide operations, and avoid gratuitous repartition(). Then size what remains: spark.sql.shuffle.partitions (default 200) with AQE partition coalescing enabled. The canonical worst case is MERGE INTO shuffling an entire table to update 5% of it — the low-shuffle MERGE deep dive covers that fix.

Diagnostics flags both high shuffle fetch-wait and shuffle-write explosion automatically, per stage.

Why is my Spark job spilling to disk?

Tasks succeed but run several times slower than they should. Spill means a task’s working set (sort buffers, aggregation hash maps, join build sides) exceeded its execution memory, so Spark serialized it to disk mid-computation and read it back. The task survives — it just pays disk I/O and serialization tax, sometimes multiple times per row.

Confirm it in the Stages tab: the Spill (Memory) and Spill (Disk) columns — they only appear when a stage actually spilled, so their presence is the diagnosis. Large spill with modest input means too few partitions: each task is chewing more data than its memory slice allows.

Fixes: raise spark.sql.shuffle.partitions so each task handles less data, or give tasks more memory — larger spark.executor.memory, fewer spark.executor.cores per executor so fewer tasks share the pool. When memory pressure escalates from spill to outright OOM kills — especially with native, off-heap engines — the sizing rules change; the accelerator OOM deep dive explains why.

Live Monitor charts spill in real time, and Diagnostics raises a memory/disk spill alert with the offending stage.

Why is GC killing my Spark job?

Executors freeze for seconds at a time; tasks show wildly inconsistent durations; heartbeats time out. JVM garbage collection pauses stop all task threads on the executor. Heavy object churn — wide rows, large aggregation maps, cached RDDs pushing heap occupancy high — makes full GCs frequent and long.

Confirm it in the Executors tab: compare GC Time to Task Time. The UI highlights the cell red past ~10%. Anything above that is runtime you’re donating to the JVM; 30%+ means the job is mostly collecting garbage.

Fixes: prefer several smaller executors over one giant heap (full-GC pause time scales with heap size), reduce caching or use serialized storage levels, and move weight off-heap (spark.memory.offHeap.enabled=true). Note that GC overhead isn’t just latency — it’s compute you pay for; the cost deep dive shows how to quantify it in executor-seconds.

Live Monitor tracks per-executor GC pressure and heap at 2-second granularity, and Diagnostics raises a GC-pressure alert before executors start dying.

Why do small files make Spark slow?

The query reads a modest amount of data but takes forever to plan and launch, with thousands of sub-second tasks. Streaming ingest and frequent commits leave lake tables with tens of thousands of tiny files. Every file costs a metadata operation, an open, and scheduling overhead — at 100k files, the fixed per-file cost dwarfs the actual read.

Confirm it in the SQL tab: open the scan node and check “number of files read” against total scan size. A 50 GB scan across 80,000 files averages ~600 KB per file — pathological. The Stages tab shows the mirror image: a huge task count where each task finishes in milliseconds.

The read-side band-aid is packing more files per task via spark.sql.files.maxPartitionBytes. The real fix is fixing the table: compact and cluster it so files are large and range-tight. That maintenance pass is itself a Spark job worth accelerating — Quanton runs it 4× faster (clustering deep dive above).

Diagnostics flags partition sizing directly: too many small partitions or too few large ones.

Why did Spark pick a bad join strategy?

A join that should broadcast a 50 MB dimension table instead sort-merge-joins it against 2 TB of facts — full shuffle and sort on both sides. Spark picks join strategies from size estimates; when table statistics are stale or missing, it overestimates the small side and falls back to SortMergeJoin.

Confirm it in the SQL tab plan graph (or EXPLAIN FORMATTED): look at the join node — BroadcastHashJoin, SortMergeJoin, or ShuffledHashJoin — and the estimated vs. actual row counts feeding it. The SHJ vs SMJ deep dive covers when each strategy actually wins.

Fixes: refresh stats (ANALYZE TABLE t COMPUTE STATISTICS), hint the join explicitly, and let AQE convert eligible joins at runtime:

SELECT /*+ BROADCAST(dim) */ * FROM facts JOIN dim ON facts.k = dim.k

Also check what reaches the join at all — Bloom filters can prune the big side before the join instead of after.

The agent’s chat has the full SQL plan in context: ask it why a join wasn’t broadcast and it answers from the actual plan, not a guess.

How do I debug a slow Spark job step by step?

The manual loop above — stage, metric, cause, config — is exactly what the Quanton AI agent automates. It’s embedded in the Spark Web UI with the full job context (stages, tasks, SQL plans, executor logs, alerts), and it requires no job code changes:

1. Open the Spark UI for the running or completed job and click AI Agent.
2. Run /spark-debugger. The agent triages the run: slowest stages, skew ratios, spill, GC pressure, shuffle volumes, straggler tasks.
3. Check the Diagnostics panel. Alerts are already raised for what the run actually hit — data skew, spill, GC pressure, OOM failures, stragglers, shuffle fetch-wait, partition sizing.
4. Click “Ask Agent” on an alert. The agent explains the mechanism using this job’s numbers and identifies the offending stage and operator.
5. Copy the fix snippet. The agent produces a ready-to-apply config or SQL change scoped to the diagnosis — not a generic tuning checklist.
6. Re-run and compare. Check the new run against the old one in the Spark History Server — stage durations and shuffle volumes tell you whether the fix landed. To make that comparison systematic (and catch the day a job quietly gets slower), build a P50/P95 baseline from your run history — the performance regressions guide shows how.

Use /performance-tuner for the proactive version of the same loop, and @-mentions to scope the agent to a specific stage or executor.

How do I measure this on my own jobs?

Run Spark Analyzer, Onehouse’s free CLI, against your Spark History Server. It reads completed runs and produces a per-application report: how much of your allocated compute did useful work, how much sat idle, which stages dominated, and what each stage was doing (extract, transform, or load) — the diagnosis half of Spark performance tuning, automated.

pip install spark-analyzer
spark-analyzer configure          # point it at your Spark History Server
spark-analyzer analyze --save-local
spark-analyzer report --input spark_analysis_output/<app>.json

--save-local keeps all data on your machine — nothing leaves your environment. We ran it against deliberately broken jobs (skew, spill, shuffle, small files) in the Spark Analyzer walkthrough if you want to see real output before pointing it at production.

Takeaway

Slow Spark jobs are rarely mysterious — they’re skewed, shuffling, spilling, collecting garbage, drowning in small files, or executing a bad plan. Diagnose first, then apply the one fix that matters.

And ceilings matter: tuning recovers waste, but the engine sets the baseline. On TPC-DS 10 TB (all 99 queries, same 11-node cluster), Quanton runs in 2,384 s versus 12,200 s for OSS Apache Spark — over 5× faster — with ~7× lower scan time on q23 from native vectorized reads. Same Spark APIs, no job code changes, and the AI agent watching every run at a 2-second refresh.

Open your slowest job’s Spark UI and ask the agent why.