Eliminated idle CPU cycles during hash-table probe and accelerated the HashAggregate operator to cut TPC-DS 10TB runtime, plus Spark Agent context enrichment, cost visibility, zero-config plugin wiring, and the option to run OSS Gluten/Comet accelerators on the operator.
Features
-
Eliminated idle CPU cycles during hash-table probe and accelerated the HashAggregate operator. On TPC-DS 10TB with 11 m8gd.4xlarge instances, Quanton now finishes in 2,384s — about 4.8% faster than the previous release (2,505s). For a comparison with OSS Spark accelerators and Databricks Photon, see the benchmark numbers.
-
Spark Agent now adds Spark History Server context enrichment (run correlation, baseline diffs, regression attribution, and failure analysis) and cost visibility — a Cost tab, per-stage cost drilldown, dynamic-allocation churn detection, and savings projection.
-
Spark Agent plugin now wires itself in automatically — set
spark.quanton.agent.enabled=trueand the agent registers intospark.pluginswithout any extra plugin configuration, on both managed and operator deployments. -
Run OSS Spark accelerators on the Quanton Operator — set
spark.quanton.accelerator=gluten(Apache Gluten) orspark.quanton.accelerator=comet(Apache DataFusion Comet) per Spark job and the operator selects the matching accelerator image. Both run on the publicapache/sparkbase with Hudi, Iceberg, and Delta plus S3/Azure/GCS connectors bundled in, and ships with the Spark Agent as well. When the flag is unset, jobs fall back to the native Quanton image.