Today we’re launching Quanton — a compute engine built to make Apache Spark fast.
Spark is the backbone of countless data pipelines. But as workloads grow, performance bottlenecks are inevitable. We built Quanton to fix that without asking you to change your code.
What is Quanton?
Quanton is a drop-in replacement for the Spark execution engine. It uses vectorized, columnar processing and adaptive query optimization to deliver significantly faster query execution on your existing Spark workloads.
How it works
Quanton slots in underneath your jobs and replaces the execution layer — your code, job definitions, catalogs, and orchestration stay exactly where they are. Three things make it fast:
- SIMD-vectorized execution. Instead of processing one row at a time, Quanton runs queries over columnar batches using SIMD instructions, so each CPU cycle does far more work than the open-source engine.
- Storage-aware query planning. Quanton understands your lakehouse tables — Apache Iceberg and Apache Hudi — and plans around them: pruning files, skipping data, and scaling work with what actually changed rather than with the size of the table.
- Background indexing. Quanton continuously builds and maintains indexes on your data, so repeat and incremental workloads get progressively faster without any extra tuning from you.
It runs as a Kubernetes-native operator on any conformant cluster — EKS, GKE, AKS, or local minikube — and your existing Spark manifests and configs work without modification. No rewrites, no new syntax, no data migration.
What’s next
Head to the Quanton docs to get started — the quickstart walks you through deploying on Kubernetes and submitting your first job.