Scribd
Upload
2 views2 pages

Apache Airflow Install

Uploaded by

Utkarsh Tiwari
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views2 pages

Apache Airflow Install

Uploaded by

Utkarsh Tiwari
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

apache_airflow_install.

md 2024-01-15

Remove existing airflow helm chart

helm delete <release-name> -n <release_namespace>

Install new release

helm repo add apache-airflow https://airflow.apache.org


helm upgrade --install airflow apache-airflow/airflow --namespace airflow --
create-namespace

Extract configuration from the running Airflow

helm show values apache-airflow/airflow > values.yaml

Update configuration

Open the values.yaml file and make the necessary changes to the configuration.
Here are our configuration updates:

Airflow Executor: By default, the executor is set to "CeleryExecutor". In our


case we prefer to use the Kubernetes executor, so update the executor value to
"KubernetesExecutor".
# Airflow executor
# One of: LocalExecutor, LocalKubernetesExecutor, CeleryExecutor,
KubernetesExecutor, CeleryKubernetesExecutor
executor: "KubernetesExecutor"

images:
airflow:
repository: our_own_apache_airflow
tag: "1.0"
# Specifying digest takes precedence over tag.
digest: ~
pullPolicy: IfNotPresent

Create Airflow Dockerfile

#Dockerfile

FROM apache/airflow:2.6.2
1/2
apache_airflow_install.md 2024-01-15

COPY requirements.txt /

RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r


/requirements.txt

Create requirement.txt file:

apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-apache-spark

Build the image:

docker build -t our_own_apache_airflow:1.0 .

Load the created image to Kubernetes internal repository

It should be that the Docker image exists in the cluster because the image was
created on the local machine.

Update airflow installation

helm upgrade --install airflow apache-airflow/airflow --namespace airflow --


values values.yaml

Let’s check access to the web UI

kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow

Next, get to http://localhost:8080 with these creds:

User: admin
Password: admin
Now you should see Airflow webserver UI.

2/2

You might also like