# Spark Performance Tuning: Why Is Your Apache Spark Job Slow — and How Do You Fix It? > Almost every slow Apache Spark job comes down to one of six causes: data skew, shuffle, spill, GC pressure, small files, or a bad query plan. Spark performance tuning starts with identifying which one is hurting your live run — not guessing configs. The Quanton AI agent, embedded in the Spark UI, pinpoints the cause automatically from the running job. Published 2026-07-16 by Quanton Team (Quanton, by Onehouse). Canonical: https://quanton.dev/blog/spark-performance-tuning --- ## Overview **Spark performance tuning** is mostly a diagnosis problem. The fixes are well known — the hard part is figuring out *which* of the six usual suspects is hurting this particular job: data skew, shuffle, spill, GC pressure, small files, or a bad query plan. Tuning configs before you've identified the cause is how teams end up with 40-line `spark-submit` incantations that fix nothing. This page is the hub. Each section below covers one failure mode: the symptom, the mechanism, where to confirm it in the Spark UI, and the fix — with a deep dive linked for each. Start with the table. | Cause | Tell-tale symptom in the Spark UI | Quick fix | Deep dive | |---|---|---|---| | Data skew / stragglers | Stages → Summary Metrics: max task duration ≫ median | AQE skew join, salt the hot key | [Spark data skew](/blog/spark-data-skew) | | Shuffle | Huge Shuffle Read/Write on a stage; high fetch wait | Cut wide operations, size partitions | [Low-shuffle MERGE](/blog/low-shuffle-merge) | | Spill | Spill (Memory) / Spill (Disk) columns appear on a stage | More partitions or more memory per task | [Accelerator memory & OOM](/blog/native-accelerator-oom) | | GC pressure | Executors tab: GC Time red, >10% of task time | Smaller heaps, more executors, off-heap | [What GC overhead costs](/blog/databricks-cost-optimization) | | Small files | Scan node reads 100k+ files; thousands of sub-second tasks | Compact / cluster the table | [Clustering 4× faster](/blog/clustering-acceleration) | | Bad query plan | SortMergeJoin where a broadcast was expected | Join hints, refresh table stats | [SHJ vs SMJ](/blog/shuffle-hash-join-vs-sort-merge-join) | ## Why is my Spark job stuck on the last task? **199 tasks finish in a minute; task 200 runs for an hour.** That's data skew: partitioning hashed one hot key (a null, a default value, one giant customer) into a single partition, and the whole stage waits on the one task processing it. Confirm it in the **Stages tab → Summary Metrics for completed tasks**. Healthy stages show max duration within ~2× of the 75th percentile. Skewed stages show a max that's 10–100× the median, and the sorted task table shows the same task with an outsized shuffle read size. The fix, in order of effort: enable AQE skew handling (on by default in Spark 3.2+): ```properties spark.sql.adaptive.enabled=true spark.sql.adaptive.skewJoin.enabled=true ``` If AQE can't split it (skewed aggregations, exploded joins), salt the hot key — the data skew deep dive linked above walks through the pattern. The Quanton Live Monitor tracks skew ratio per stage at a 2-second refresh, and Diagnostics raises a data-skew alert the moment a straggler emerges — while the job is still running. ## Why is shuffle the bottleneck? **The stage is slow but no single task stands out — everything is waiting on the network.** Every wide operation (joins, groupBy, repartition, window functions) shuffles data between executors: serialize, write to disk, transfer, fetch, deserialize. When a job moves hundreds of GiB per shuffle, that movement *is* the runtime. Confirm it in the **Stages tab: Shuffle Read and Shuffle Write columns**. If a stage shuffles a large multiple of what it eventually outputs, the plan is moving data it doesn't need to. High **shuffle read fetch wait time** in task metrics means executors sit idle waiting on remote blocks. Fixes: eliminate shuffles before shrinking them — broadcast the small side of joins, pre-aggregate before wide operations, and avoid gratuitous `repartition()`. Then size what remains: `spark.sql.shuffle.partitions` (default 200) with AQE partition coalescing enabled. The canonical worst case is `MERGE INTO` shuffling an entire table to update 5% of it — the low-shuffle MERGE deep dive covers that fix. Diagnostics flags both high shuffle fetch-wait and shuffle-write explosion automatically, per stage. ## Why is my Spark job spilling to disk? **Tasks succeed but run several times slower than they should.** Spill means a task's working set (sort buffers, aggregation hash maps, join build sides) exceeded its execution memory, so Spark serialized it to disk mid-computation and read it back. The task survives — it just pays disk I/O and serialization tax, sometimes multiple times per row. Confirm it in the **Stages tab: the Spill (Memory) and Spill (Disk) columns** — they only appear when a stage actually spilled, so their presence is the diagnosis. Large spill with modest input means too few partitions: each task is chewing more data than its memory slice allows. Fixes: raise `spark.sql.shuffle.partitions` so each task handles less data, or give tasks more memory — larger `spark.executor.memory`, fewer `spark.executor.cores` per executor so fewer tasks share the pool. When memory pressure escalates from spill to outright OOM kills — especially with native, off-heap engines — the sizing rules change; the accelerator OOM deep dive explains why. Live Monitor charts spill in real time, and Diagnostics raises a memory/disk spill alert with the offending stage. ## Why is GC killing my Spark job? **Executors freeze for seconds at a time; tasks show wildly inconsistent durations; heartbeats time out.** JVM garbage collection pauses stop all task threads on the executor. Heavy object churn — wide rows, large aggregation maps, cached RDDs pushing heap occupancy high — makes full GCs frequent and long. Confirm it in the **Executors tab: compare GC Time to Task Time**. The UI highlights the cell red past ~10%. Anything above that is runtime you're donating to the JVM; 30%+ means the job is mostly collecting garbage. Fixes: prefer several smaller executors over one giant heap (full-GC pause time scales with heap size), reduce caching or use serialized storage levels, and move weight off-heap (`spark.memory.offHeap.enabled=true`). Note that GC overhead isn't just latency — it's compute you pay for; the cost deep dive shows how to quantify it in executor-seconds. Live Monitor tracks per-executor GC pressure and heap at 2-second granularity, and Diagnostics raises a GC-pressure alert before executors start dying. ## Why do small files make Spark slow? **The query reads a modest amount of data but takes forever to plan and launch, with thousands of sub-second tasks.** Streaming ingest and frequent commits leave lake tables with tens of thousands of tiny files. Every file costs a metadata operation, an open, and scheduling overhead — at 100k files, the fixed per-file cost dwarfs the actual read. Confirm it in the **SQL tab: open the scan node and check "number of files read"** against total scan size. A 50 GB scan across 80,000 files averages ~600 KB per file — pathological. The Stages tab shows the mirror image: a huge task count where each task finishes in milliseconds. The read-side band-aid is packing more files per task via `spark.sql.files.maxPartitionBytes`. The real fix is fixing the table: compact and cluster it so files are large and range-tight. That maintenance pass is itself a Spark job worth accelerating — Quanton runs it 4× faster (clustering deep dive above). Diagnostics flags partition sizing directly: too many small partitions or too few large ones. ## Why did Spark pick a bad join strategy? **A join that should broadcast a 50 MB dimension table instead sort-merge-joins it against 2 TB of facts** — full shuffle and sort on both sides. Spark picks join strategies from size estimates; when table statistics are stale or missing, it overestimates the small side and falls back to SortMergeJoin. Confirm it in the **SQL tab plan graph** (or `EXPLAIN FORMATTED`): look at the join node — BroadcastHashJoin, SortMergeJoin, or ShuffledHashJoin — and the estimated vs. actual row counts feeding it. The SHJ vs SMJ deep dive covers when each strategy actually wins. Fixes: refresh stats (`ANALYZE TABLE t COMPUTE STATISTICS`), hint the join explicitly, and let AQE convert eligible joins at runtime: ```sql SELECT /*+ BROADCAST(dim) */ * FROM facts JOIN dim ON facts.k = dim.k ``` Also check what reaches the join at all — [Bloom filters can prune the big side before the join](/blog/bloom-filters-before-joins) instead of after. The agent's chat has the full SQL plan in context: ask it why a join wasn't broadcast and it answers from the actual plan, not a guess. ## How do I debug a slow Spark job step by step? The manual loop above — stage, metric, cause, config — is exactly what the Quanton AI agent automates. It's embedded in the Spark Web UI with the full job context (stages, tasks, SQL plans, executor logs, alerts), and it requires no job code changes: 1. **Open the Spark UI** for the running or completed job and click **AI Agent**. 2. **Run `/spark-debugger`.** The agent triages the run: slowest stages, skew ratios, spill, GC pressure, shuffle volumes, straggler tasks. 3. **Check the Diagnostics panel.** Alerts are already raised for what the run actually hit — data skew, spill, GC pressure, OOM failures, stragglers, shuffle fetch-wait, partition sizing. 4. **Click "Ask Agent" on an alert.** The agent explains the mechanism using this job's numbers and identifies the offending stage and operator. 5. **Copy the fix snippet.** The agent produces a ready-to-apply config or SQL change scoped to the diagnosis — not a generic tuning checklist. 6. **Re-run and compare.** Check the new run against the old one in the Spark History Server — stage durations and shuffle volumes tell you whether the fix landed. To make that comparison systematic (and catch the day a job quietly gets slower), build a P50/P95 baseline from your run history — the [performance regressions guide](/blog/spark-performance-regressions) shows how. Use `/performance-tuner` for the proactive version of the same loop, and `@`-mentions to scope the agent to a specific stage or executor. ## How do I measure this on my own jobs? Run [Spark Analyzer](https://pypi.org/project/spark-analyzer/), Onehouse's free CLI, against your Spark History Server. It reads completed runs and produces a per-application report: how much of your allocated compute did useful work, how much sat idle, which stages dominated, and what each stage was doing (extract, transform, or load) — the diagnosis half of Spark performance tuning, automated. ```bash pip install spark-analyzer spark-analyzer configure # point it at your Spark History Server spark-analyzer analyze --save-local spark-analyzer report --input spark_analysis_output/.json ``` `--save-local` keeps all data on your machine — nothing leaves your environment. We ran it against deliberately broken jobs (skew, spill, shuffle, small files) in the [Spark Analyzer walkthrough](/blog/spark-analyzer) if you want to see real output before pointing it at production. ## Takeaway Slow Spark jobs are rarely mysterious — they're skewed, shuffling, spilling, collecting garbage, drowning in small files, or executing a bad plan. Diagnose first, then apply the one fix that matters. And ceilings matter: tuning recovers waste, but the engine sets the baseline. On TPC-DS 10 TB (all 99 queries, same 11-node cluster), Quanton runs in 2,384 s versus 12,200 s for OSS Apache Spark — over 5× faster — with ~7× lower scan time on q23 from native vectorized reads. Same Spark APIs, no job code changes, and the AI agent watching every run at a 2-second refresh. **Open your slowest job's Spark UI and ask the agent why.** ## Frequently asked questions ### Why is my Spark job suddenly slow? A job that was fast yesterday and slow today usually changed one of three things: input data (more volume, new skew, more small files), the query plan (stats went stale, a join flipped from broadcast to sort-merge), or the cluster (fewer executors, noisy neighbors). Compare the slow run against a known-good run stage by stage instead of guessing — on Quanton, give the slow run to the AI agent and it attributes the slowdown to specific stages from the task metrics. ### How do I debug a slow Spark job? Find the slowest stage in the Spark UI, then check its Summary Metrics: a max task duration far above the median means skew, Spill columns mean memory pressure, huge Shuffle Read/Write means a shuffle bottleneck, and high GC Time on the Executors tab means heap pressure. Fix the one dominant cause first. With Quanton, run /spark-debugger in the AI agent inside the Spark UI and it walks this triage for you. ### Why is my Spark job stuck at the last task? One partition holds far more data than the rest — data skew — so one task keeps running after the other 199 finish. Confirm it in the Stages tab: the max task duration will be many times the 75th percentile. Enable AQE skew-join handling (spark.sql.adaptive.skewJoin.enabled=true) or salt the hot key. ### Why does one Spark task take forever? Either that task's partition is oversized (data skew) or its executor is unhealthy (GC thrashing, spilling to disk, or a slow node). Check the task's shuffle read size versus its peers: if it read much more data, it's skew; if the data size is normal but the duration is not, look at that executor's GC time and spill metrics. ### How do I know which join strategy Spark used? Open the SQL / DataFrame tab in the Spark UI and inspect the plan graph, or run EXPLAIN FORMATTED on the query. You'll see BroadcastHashJoin, SortMergeJoin, or ShuffledHashJoin as the join node. A SortMergeJoin against a table small enough to broadcast is a classic sign of stale statistics. ### What are straggler tasks in Spark? Stragglers are tasks that run dramatically longer than the other tasks in the same stage, so the whole stage waits on them. They're caused by data skew, degraded executors, or GC pauses. The Quanton Live Monitor tracks task-duration stragglers continuously and Diagnostics raises a straggler-task alert when one appears. ### Does Spark performance tuning require code changes? Often not. Skew handling, partition coalescing, broadcast thresholds, and memory sizing are all config-level fixes, and AQE handles several of them automatically in Spark 3.x. Code changes come in for key salting and query restructuring. Quanton's engine-level speedups and its AI agent both work without touching job code.