From the team.
Why Native Spark Accelerators Get OOMKilled, and How Quanton Runs Reliably Under Memory Pressure
Native Spark accelerators move most of a query's working set out of the JVM heap and into off-heap native memory — and that is exactly where they tend to get OOMKilled. The surprising part is that the engine often frees its memory correctly; the kill happens anyway. Quanton is built to track live memory accurately, avoid fragmentation and spill under pressure instead of dying.
Bloom Filters Before the Join: How Spark Prunes Probe Rows — and How Quanton Makes It Native
Spark applies Bloom filters to large fact tables before joining them with dimension tables, reducing the amount of data that must be shuffled and joined. Quanton preserves Spark's Bloom filter injection strategy while accelerating filter evaluation through vectorized execution.
GROUP BY ROLLUP Without the Row Explosion
ROLLUP is how SQL computes subtotals and grand totals in one query. Spark executes it by exploding every input row into N+1 copies before aggregating — a tax that grows with the depth of the rollup. Quanton replaces that with a rollup operator that re-aggregates already-collapsed state, removing the input-side explosion.
Real Apache Spark. Inside Snowflake.
We talked with 1,200+ data engineers at Snowflake Summit and Databricks Data+AI Summit. Only 15% of Snowflake users are on Iceberg. We think that's about to change. Today we're shipping Quanton on SPCS: real Apache Spark inside your Snowflake account, 2-5x better price/performance than Databricks with Photon, 63% fewer credits burned.
Shuffle Hash Join vs. Sort-Merge Join: How Modern Engines Execute Equi-Joins
SHJ and SMJ produce identical results but make opposite bets on CPU, memory, and data layout. A deep dive into how vectorized engines implement both — adaptive hash-table layouts, SIMD tag probing, normalized keys — and why Quanton defaults to a vectorized SHJ.
Accelerating Lakehouse Reads with Quanton Engine
High and unpredictable object-store access latencies, compute-intensive Parquet decoding, and shuffle preparation can dominate Spark job runtimes. Quanton is built to optimize this critical first mile.
Why Clustering Matters — and Why We Just Made It 4× Faster
Clustering fixes the two file-layout problems every data team hits: too many files scanned, and small files piling up. The catch is that the sort-and-rewrite isn't free. Quanton ships a native execution path for clustering on Hudi and Iceberg.
MERGE INTO Updates a Slice of Your Table — So Why Shuffle All of It?
A typical MERGE updates a tiny fraction of rows, yet OSS Apache Spark + Apache Iceberg shuffles the entire target table. Here's the shape of that pain — and the low-shuffle columnar MERGE path Quanton uses instead.
Introducing Quanton: Spark at Full Speed
Today we're launching Quanton, a drop-in compute engine that makes your Apache Spark workloads dramatically faster — no rewrites required.
No posts in this category yet.