Quanton Operator Releases.

Improvements

• TPC-DS 10 TB in 2,034 seconds. On 11 m8gd.4xlarge instances, Quanton now finishes the suite in 2,034s, about 9.5% faster than the previous release (2,247s). For a comparison with OSS Spark accelerators and Databricks Photon, see the benchmark numbers.
• Faster scan planning on tables with large Parquet footers. Quanton reads large footers with fewer object-storage requests and keeps scans parallel across the cluster.
• Faster native Hudi and Iceberg reads with hudi-rs and iceberg-rust. The native Hudi reader picks up the latest hudi-rs work, and the Spark 4.1 image reads Iceberg natively through iceberg-rust:
  • Merge-on-read snapshot queries use file group reader v2, which is now the default.
  • Base-file reads prune Parquet row groups with the query’s predicates. A merge-aware gate keeps a predicate out of the base read when pushing it down could drop a record that a log file updates.
  • Base-file reads start with a bounded prefetch from object storage, and the reader reuses one object-store client per bucket and set of options instead of building a new one for each read.
  • Metadata-table reads go through the same reader, sharded and memory-bounded, and copy only the HFile record they need instead of the whole block.
  • Schema evolution fills defaults for date, time, decimal, fixed, and enum fields, and resolves older HFile schemas to the reader schema.
  • Iceberg v3 tables with deletion vectors are read natively through iceberg-rust, including row lineage (_row_id and _last_updated_sequence_number).
• Aggregation pushdown through joins. Quanton can pre-aggregate rows below a join, so the join and its shuffle process fewer rows.

Features

• Iceberg on Spark 4.1. Accelerated compaction and MERGE INTO now support deletion vectors and VARIANT columns, with 4–5× faster compaction and merges compared with the open-source baseline. The operator automatically matches jobs that specify Spark 4.x to the Spark 4.1 image.

• Accelerated compaction for VARIANT data. Quanton compacts VARIANT columns with its columnar approach, avoiding unnecessary decoding and re-encoding.

• Native Hudi reads with hudi-rs. Hudi 1.x images now accelerate merge-on-read snapshot queries alongside copy-on-write queries. The Rust reader combines base files with updates from delta logs, bringing native execution to tables with log files.

• Read documents as Spark tables. The bundled quanton_unstructured data source extracts text and chunks from PDF, HTML, Office, EPUB, text and Markdown files. Each row includes file metadata and parse status, so a document that fails to parse does not stop the job.

• Export fine-tuning datasets. The bundled training_dataset writer validates and exports curated DataFrames for Together, Fireworks or Baseten, with a manifest and provider-specific loading instructions. It prepares the dataset in object storage; you control when to upload it and start training.

• Comet 1.0.0 on Spark 4.1. The operator adds a separate Spark 4.1 image for Apache DataFusion Comet, alongside its existing Spark 3.5 accelerator images.

Improvements

• Improved HashShuffleWriter performance for aggregation workloads by reducing per-row serialization overhead during shuffle writes and deserialization overhead during shuffle reads.
• Aligned shuffle size tracking with skew-join detection to identify and split oversized partitions.
• Runtime images now include boto3, numpy, pandas, and operational CLI tools.

Features

• Eliminated idle CPU cycles during hash-table probe and accelerated the HashAggregate operator. On TPC-DS 10TB with 11 m8gd.4xlarge instances, Quanton now finishes in 2,384s — about 4.8% faster than the previous release (2,505s). For a comparison with OSS Spark accelerators and Databricks Photon, see the benchmark numbers.

• Spark Agent now adds Spark History Server context enrichment (run correlation, baseline diffs, regression attribution, and failure analysis) and cost visibility — a Cost tab, per-stage cost drilldown, dynamic-allocation churn detection, and savings projection.

• Spark Agent plugin now wires itself in automatically — set spark.quanton.agent.enabled=true and the agent registers into spark.plugins without any extra plugin configuration, on both managed and operator deployments.

• Run OSS Spark accelerators on the Quanton Operator — set spark.quanton.accelerator=gluten (Apache Gluten) or spark.quanton.accelerator=comet (Apache DataFusion Comet) per Spark job and the operator selects the matching accelerator image. Both run on the public apache/spark base with Hudi, Iceberg, and Delta plus S3/Azure/GCS connectors bundled in, and ships with the Spark Agent as well. When the flag is unset, jobs fall back to the native Quanton image.

Features

• Native ROLLUP operator runs GROUP BY ROLLUP in a single pass without row explosion — up to 5× faster than open-source Spark.
• Automatic self-join elimination enabled by default.
• Catalog caching across Iceberg, Hudi, Glue, and Onehouse catalogs to cut repeated metadata lookups.

Improvements

• Optimized table scan operator that’s column-size aware.
• JSON-formatted driver pod logs.

Features

• Spark Agent improvements and patches to reduce hallucinations.
• Iceberg vended credentials with the native object-storage client for S3, GCS, and ADLS — works with any Iceberg REST Catalog, including AWS Glue, Snowflake Polaris, and Databricks Unity.

Features

• Quanton-native clustering accelerator — 4× faster than OSS.
• AI agent tool framework.
• Conditional-simplification query optimization.

Improvements

• In-progress queries now reported through the /metrics/sql endpoint.

Highlights

• Initial production release of the Quanton Operator Spark image — the vectorized, Velox-backed Spark execution engine packaged as a Kubernetes-native operator.