AWS

Deploy the Quanton Operator on Amazon Web Services using EKS.

tip

New to EKS? Follow the EKS deployment guide for a step-by-step walkthrough from cluster creation to your first Spark job.

EKS

Amazon Elastic Kubernetes Service (EKS) is the recommended deployment target for Quanton on AWS. The Quanton Operator runs on your EKS cluster and manages the full Spark job lifecycle via Kubernetes.

Prerequisites

• EKS cluster running Kubernetes >= 1.28
• Helm >= 3.x and kubectl configured for your cluster
• onehouse-values.yaml downloaded from the Onehouse console
• Outbound network access from your cluster to *.onehouse.ai and *.docker.io

Step 1: Install the Spark Operator

The Quanton Operator builds on top of the kubeflow Spark Operator. Install it first:

helm repo add spark-operator https://kubeflow.github.io/spark-operator
helm repo update

helm install spark-operator spark-operator/spark-operator \
  --namespace spark-operator \
  --create-namespace \
  --set "spark.jobNamespaces={default}"

Verify it's running:

kubectl get pods -n spark-operator

Step 2: Install the Quanton Operator

helm upgrade --install quanton-operator oci://registry-1.docker.io/onehouseai/quanton-operator \
  --namespace quanton-operator \
  --create-namespace \
  --set "quantonOperator.jobNamespaces={default}" \
  -f /path/to/onehouse-values.yaml

Verify the operator pod is running:

kubectl get pods -n quanton-operator

Step 3: Submit a Spark job

apiVersion: quantonsparkoperator.onehouse.ai/v1beta2
kind: QuantonSparkApplication
metadata:
  name: my-spark-job
  namespace: default
spec:
  sparkApplicationSpec:
    type: Python
    mode: cluster
    image: "dist.onehouse.ai/onehouseai/quanton-spark:release-v1.29.0-al2023"
    mainApplicationFile: "s3://my-bucket/jobs/my_job.py"
    sparkVersion: "3.5.0"
    sparkConf:
      "spark.hadoop.fs.s3a.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenFileCredentialsProvider"
    driver:
      cores: 4
      memory: "8192m"
      serviceAccount: spark-operator-spark
    executor:
      cores: 4
      instances: 4
      memory: "8192m"

kubectl apply -f my-spark-job.yaml

S3 access via IRSA

Use IRSA (IAM Roles for Service Accounts) to give driver and executor pods access to S3 without static credentials.

1. Associate an OIDC provider with your cluster (one-time, per cluster):

eksctl utils associate-iam-oidc-provider \
  --cluster <cluster-name> \
  --region <region> \
  --approve

2. Create the IAM role and bind it to the service account in one step. Replace <policy-arn> with a policy granting s3:GetObject, s3:PutObject, and s3:ListBucket on your data buckets:

eksctl create iamserviceaccount \
  --name spark-operator-spark \
  --namespace default \
  --cluster <cluster-name> \
  --region <region> \
  --attach-policy-arn <policy-arn> \
  --approve \
  --override-existing-serviceaccounts

This creates the IAM role, scopes its trust policy to system:serviceaccount:default:spark-operator-spark, and annotates the service account with the role ARN. If you'd rather create the role manually, annotate the existing service account instead:

kubectl annotate serviceaccount spark-operator-spark \
  eks.amazonaws.com/role-arn=arn:aws:iam::<account>:role/SparkS3Role \
  -n default

The role's trust policy must federate with the cluster's OIDC provider and be scoped to system:serviceaccount:default:spark-operator-spark — otherwise pods get 403s on S3 at runtime with no clear error.

Dedicated node group (optional)

For best performance, run Spark pods on a dedicated node group:

eksctl create nodegroup \
  --cluster my-cluster \
  --name spark-workers \
  --node-type m5.2xlarge \
  --nodes 4 \
  --node-labels workload=spark

Set a matching node selector in onehouse-values.yaml:

quantonOperator:
  nodeSelector:
    workload: spark

Then re-apply the Helm install with the updated values file.