The Quanton Blog
Engineering deep-dives, guides, and product news from the people building Quanton.
Reducing Spark SQL Shuffle with Partial Aggregation Before Joins
Quanton calculates partial sums and counts before eligible joins, so the join processes fewer rows. Our reported TPC-DS results at 10 TB show a 221-second reduction in total runtime.
Fine-Tuning Open Models: From Lakehouse Tables to a Training Dataset
Fine-tuning changes how a model behaves, and RAG changes what it answers from. This guide covers what fine-tuning is and the kinds you will meet: LoRA against full weights, and SFT against preference tuning and continued pretraining. It then covers where you run the job, either with an open-source trainer on your own GPUs or on a managed platform. Finally it builds the dataset itself in SQL from the Hudi tables that the previous guide produced. The exporter validates that dataset against the trainer's schema before anything is written, versions it so that every run stays reproducible, and writes a manifest naming the exact command that loads it.
RAG on Your Lakehouse: Serve Context From Structured and Unstructured Data in SQL
Your object storage holds structured data such as fact tables, exports and CDC feeds, and unstructured data such as PDF, TXT, MD and HTML files. Many teams keep those files on a NAS and read them through an object-storage client. This guide reads both kinds with one engine, and it parses contracts from the CUAD dataset on Hugging Face into Hudi tables that hold the chunks and the embeddings. One SQL query then answers a question, mixing filter, join and aggregate with vector similarity in the same plan.
Index-Aware MERGE: Reading Only the Row Groups That Match
A JOIN or MERGE has to find the target rows that match your source keys. Spark tries to prune the target with dynamic partition pruning, but when updates land randomly across a large fact table it barely helps, so it falls back to scanning and shuffling most of the table. Quanton asks an index instead, mapping each key to the exact file and row position so the engine reads only the row groups that actually match.
Why Native Spark Accelerators Get OOMKilled Under Memory Pressure
Native Spark accelerators move most of a query's working set out of the JVM heap and into off-heap native memory — and that is exactly where they tend to get OOMKilled. The surprising part is that the engine often frees its memory correctly; the kill happens anyway. Quanton is built to track live memory accurately, avoid fragmentation and spill under pressure instead of dying.
Runtime Bloom Filters in Spark Joins: How Probe-Side Pruning Works
Spark applies Bloom filters to large fact tables before joining them with dimension tables, reducing the amount of data that must be shuffled and joined. Quanton preserves Spark's Bloom filter injection strategy while accelerating filter evaluation through vectorized execution.
GROUP BY ROLLUP Without the Row Explosion
ROLLUP is how SQL computes subtotals and grand totals in one query. Spark executes it by exploding every input row into N+1 copies before aggregating — a tax that grows with the depth of the rollup. Quanton replaces that with a rollup operator that re-aggregates already-collapsed state, removing the input-side explosion.
Running Apache Spark on Snowflake Container Services (SPCS)
We talked with 1,200+ data engineers at Snowflake Summit and Databricks Data+AI Summit. Only 15% of Snowflake users are on Iceberg. We think that's about to change. Today we're shipping Quanton on SPCS: real Apache Spark inside your Snowflake account, 2-5x better price/performance than Databricks with Photon, 63% fewer credits burned.
Shuffle Hash Join vs. Sort-Merge Join: How Modern Engines Execute Equi-Joins
SHJ and SMJ produce identical results but make opposite bets on CPU, memory, and data layout. A deep dive into how vectorized engines implement both — adaptive hash-table layouts, SIMD tag probing, normalized keys — and why Quanton defaults to a vectorized SHJ.
Speeding Up Lakehouse Reads: Object-Store Latency, Parquet Decoding and Shuffle
High and unpredictable object-store access latencies, compute-intensive Parquet decoding, and shuffle preparation can dominate Spark job runtimes. Quanton is built to optimize this critical first mile.
Native Clustering for Hudi and Iceberg Tables
Clustering fixes the two file-layout problems every data team hits: too many files scanned, and small files piling up. The catch is that the sort-and-rewrite isn't free. Quanton ships a native execution path for clustering on Hudi and Iceberg.
Low-Shuffle MERGE INTO for Apache Iceberg
A typical MERGE updates a tiny fraction of rows, yet OSS Apache Spark + Apache Iceberg shuffles the entire target table. Here's the shape of that pain — and the low-shuffle columnar MERGE path Quanton uses instead.
Introducing Quanton: Spark at Full Speed
Today we're launching Quanton, a drop-in compute engine that makes your Apache Spark workloads dramatically faster — no rewrites required.
No posts match .
