Azure

Deploy the Quanton Operator on Microsoft Azure using AKS.

tip

New to AKS? Follow the AKS deployment guide for a step-by-step walkthrough from cluster creation to your first Spark job.

AKS

Azure Kubernetes Service (AKS) is the recommended deployment target for Quanton on Azure. The Quanton Operator runs on your AKS cluster and manages the full Spark job lifecycle via Kubernetes.

Prerequisites

• AKS cluster running Kubernetes >= 1.28
• Helm >= 3.x and kubectl configured for your cluster
• onehouse-values.yaml downloaded from the Onehouse console
• Outbound network access from your cluster to *.onehouse.ai and *.docker.io

Step 1: Install the Spark Operator

The Quanton Operator builds on top of the kubeflow Spark Operator. Install it first:

helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm repo update

helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \
  --create-namespace \
  --set "spark.jobNamespaces={default}"

Verify it's running:

kubectl get pods -n spark-operator

Step 2: Install the Quanton Operator

helm upgrade --install quanton-operator oci://registry-1.docker.io/onehouseai/quanton-operator \
  --namespace quanton-operator \
  --create-namespace \
  --set "quantonOperator.jobNamespaces={default}" \
  -f onehouse-values.yaml

Verify the operator pod is running:

kubectl get pods -n quanton-operator

Step 3: Submit a Spark job

apiVersion: quantonsparkoperator.onehouse.ai/v1beta2
kind: QuantonSparkApplication
metadata:
  name: my-spark-job
  namespace: default
spec:
  sparkApplicationSpec:
    type: Python
    mode: cluster
    image: "dist.onehouse.ai/onehouseai/quanton-spark:release-v1.29.0-al2023"
    mainApplicationFile: "abfss://my-container@mystorageaccount.dfs.core.windows.net/jobs/my_job.py"
    sparkVersion: "3.5.0"
    sparkConf:
      "spark.hadoop.fs.azure.account.auth.type.mystorageaccount.dfs.core.windows.net": "OAuth"
      "spark.hadoop.fs.azure.account.oauth.provider.type.mystorageaccount.dfs.core.windows.net": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
      "spark.hadoop.fs.azure.account.oauth2.client.id.mystorageaccount.dfs.core.windows.net": "<client-id>"
      "spark.hadoop.fs.azure.account.oauth2.client.secret.mystorageaccount.dfs.core.windows.net": "<client-secret>"
      "spark.hadoop.fs.azure.account.oauth2.client.endpoint.mystorageaccount.dfs.core.windows.net": "https://login.microsoftonline.com/<tenant-id>/oauth2/token"
    driver:
      cores: 4
      memory: "8192m"
      serviceAccount: spark-operator-spark
    executor:
      cores: 4
      instances: 4
      memory: "8192m"

kubectl apply -f my-spark-job.yaml

ADLS Gen2 access via Service Principal

Store the Service Principal credentials in a Kubernetes Secret:

kubectl create secret generic adls-sp \
  --from-literal=client-id=<client-id> \
  --from-literal=client-secret=<client-secret> \
  -n default

Reference the secret values via environment variables in the job spec and use ${SP_CLIENT_ID} / ${SP_CLIENT_SECRET} in your Spark config values.

Dedicated node pool (optional)

For best performance, run Spark pods on a dedicated node pool:

az aks nodepool add \
  --resource-group my-rg \
  --cluster-name my-cluster \
  --name sparkpool \
  --node-vm-size Standard_D8s_v3 \
  --node-count 4 \
  --labels workload=spark

Set a matching node selector in onehouse-values.yaml:

quantonOperator:
  nodeSelector:
    workload: spark

Then re-apply the Helm install with the updated values file.