Quanton Operator Releases.

Features

• Iceberg on Spark 4.1. Accelerated compaction and MERGE INTO now support deletion vectors and VARIANT columns, with 4–5× faster compaction and merges compared with the open-source baseline. The operator automatically matches jobs that specify Spark 4.x to the Spark 4.1 image.

• Accelerated compaction for VARIANT data. Quanton compacts VARIANT columns with its columnar approach, avoiding unnecessary decoding and re-encoding.

• Native Hudi reads with hudi-rs. Hudi 1.x images now accelerate merge-on-read snapshot queries alongside copy-on-write queries. The Rust reader combines base files with updates from delta logs, bringing native execution to tables with log files.

• Read documents as Spark tables. The bundled quanton_unstructured data source extracts text and chunks from PDF, HTML, Office, EPUB, text and Markdown files. Each row includes file metadata and parse status, so a document that fails to parse does not stop the job.

• Export fine-tuning datasets. The bundled training_dataset writer validates and exports curated DataFrames for Together, Fireworks or Baseten, with a manifest and provider-specific loading instructions. It prepares the dataset in object storage; you control when to upload it and start training.

• Comet 1.0.0 on Spark 4.1. The operator adds a separate Spark 4.1 image for Apache DataFusion Comet, alongside its existing Spark 3.5 accelerator images.

Improvements

• Improved HashShuffleWriter performance for aggregation workloads by reducing per-row serialization overhead during shuffle writes and deserialization overhead during shuffle reads.
• Aligned shuffle size tracking with skew-join detection to identify and split oversized partitions.
• Runtime images now include boto3, numpy, pandas, and operational CLI tools.

Features

• Eliminated idle CPU cycles during hash-table probe and accelerated the HashAggregate operator. On TPC-DS 10TB with 11 m8gd.4xlarge instances, Quanton now finishes in 2,384s — about 4.8% faster than the previous release (2,505s). For a comparison with OSS Spark accelerators and Databricks Photon, see the benchmark numbers.

• Spark Agent now adds Spark History Server context enrichment (run correlation, baseline diffs, regression attribution, and failure analysis) and cost visibility — a Cost tab, per-stage cost drilldown, dynamic-allocation churn detection, and savings projection.

• Spark Agent plugin now wires itself in automatically — set spark.quanton.agent.enabled=true and the agent registers into spark.plugins without any extra plugin configuration, on both managed and operator deployments.

• Run OSS Spark accelerators on the Quanton Operator — set spark.quanton.accelerator=gluten (Apache Gluten) or spark.quanton.accelerator=comet (Apache DataFusion Comet) per Spark job and the operator selects the matching accelerator image. Both run on the public apache/spark base with Hudi, Iceberg, and Delta plus S3/Azure/GCS connectors bundled in, and ships with the Spark Agent as well. When the flag is unset, jobs fall back to the native Quanton image.

Features

• Native ROLLUP operator runs GROUP BY ROLLUP in a single pass without row explosion — up to 5× faster than open-source Spark.
• Automatic self-join elimination enabled by default.
• Catalog caching across Iceberg, Hudi, Glue, and Onehouse catalogs to cut repeated metadata lookups.

Improvements

• Optimized table scan operator that’s column-size aware.
• JSON-formatted driver pod logs.

Features

• Spark Agent improvements and patches to reduce hallucinations.
• Iceberg vended credentials with the native object-storage client for S3, GCS, and ADLS — works with any Iceberg REST Catalog, including AWS Glue, Snowflake Polaris, and Databricks Unity.

Features

• Quanton-native clustering accelerator — 4× faster than OSS.
• AI agent tool framework.
• Conditional-simplification query optimization.

Improvements

• In-progress queries now reported through the /metrics/sql endpoint.

Highlights

• Initial production release of the Quanton Operator Spark image — the vectorized, Velox-backed Spark execution engine packaged as a Kubernetes-native operator.