Skip to content
This repository was archived by the owner on Jan 28, 2022. It is now read-only.

Latest commit

 

History

History
105 lines (70 loc) · 3.02 KB

deploy.md

File metadata and controls

105 lines (70 loc) · 3.02 KB

Deploy the operator

Prerequests

  • You have kubectl configured pointing to the target Kubernetes cluster.
  • You have access to a DataBricks cluster and able to generate PAT token. To generate a token, check generate a DataBricks token.

Step-by-step guide

This will deploy the operator in namespace azure-databricks-operator-system. If you want to customise the namespace, you can either search-replace the namespace, or use kustomise by following the next section.

  1. Download the latest release manifests:
wget https://github.com/microsoft/azure-databricks-operator/releases/latest/download/release.zip
unzip release.zip

(optional) Configure maximum number of run reconcilers

  1. Create the azure-databricks-operator-system namespace:
kubectl create namespace azure-databricks-operator-system
  1. Create Kubernetes secrets with values for DATABRICKS_HOST and DATABRICKS_TOKEN:
kubectl --namespace azure-databricks-operator-system \
    create secret generic dbrickssettings \
    --from-literal=DatabricksHost="https://xxxx.azuredatabricks.net" \
    --from-literal=DatabricksToken="xxxxx"
  1. Apply the manifests for the Operator and CRDs in release/config:
kubectl apply -f release/config

Configure maximum number of run reconcilers

  1. Change the MAX_CONCURRENT_RUN_RECONCILES value in config/default/manager_image_patch.yaml under the env section with the desired number of reconcilers
          - name: MAX_CONCURRENT_RUN_RECONCILES
            value: "1"

By default MAX_CONCURRENT_RUN_RECONCILES is set to 1

Use kustomize to customise your deployment

  1. Clone the source code:
git clone git@github.com:microsoft/azure-databricks-operator.git

  1. Edit file config/default/kustomization.yaml file to change your preferences

  2. Use kustomize to generate the final manifests and deploy:

kustomize build config/default | kubectl apply -f -
  1. Deploy the CRDs:
kubectl apply -f config/crd/bases

Test your deployment

  1. Deploy a sample job, this will create a job in the default namespace:
curl https://raw.githubusercontent.com/microsoft/azure-databricks-operator/master/config/samples/databricks_v1alpha1_djob.yaml | kubectl apply -f -
  1. Check the Job in Kubernetes:
kubectl get djob
  1. Check the job is created successfully in DataBricks.

Troubleshooting

If you encounter any issue, you can check the log of the operator by pulling it from Kubernetes:

# get the pod name of your operator
kubectl --namespace azure-databricks-operator-system get pods

# pull the logs
kubectl --namespace azure-databricks-operator-system logs -f [name_of_the_operator_pod]

To further aid debugging diagnostic metrics are produced by the operator. Please review the metrics page for further information