Apache Airflow setup on kubernetes using helm charts

[bsce20007]

Hello I am trying to install apache airflow on kubernetes using heml charts with default values.yaml configuration as given is documentation , Airflow is unable to get installed on kubernetes.
denovonet@master:~/Work/airflow-helm$ helm repo add apache-airflow https://airflow.apache.org
helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace
"apache-airflow" already exists with the same configuration, skipping
Release "airflow" does not exist. Installing it now.
Error: failed post-install: 1 error occurred:
* timed out waiting for the condition

denovonet@master:~/Work/airflow-helm$ kubectl get pods -n airflow
NAME READY STATUS RESTARTS AGE
airflow-postgresql-0 0/1 Pending 0 12m
airflow-redis-0 0/1 Pending 0 12m
airflow-run-airflow-migrations-zrst9 1/1 Running 3 (3m41s ago) 12m
airflow-scheduler-7d588dbc6-j6svn 0/2 Init:0/1 3 (3m42s ago) 12m
airflow-statsd-6df8564664-tbsbv 1/1 Running 0 12m
airflow-triggerer-0 0/2 Pending 0 12m
airflow-webserver-76b54cbbfc-x8t8p 0/1 Init:0/1 3 (3m41s ago) 12m
airflow-worker-0 0/2 Pending 0 12m

denovonet@master:/Work/airflow-helm$ kubectl get events -n airflow
LAST SEEN TYPE REASON OBJECT MESSAGE
3m2s Warning FailedScheduling pod/airflow-postgresql-0 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
23m Normal SuccessfulCreate statefulset/airflow-postgresql create Claim data-airflow-postgresql-0 Pod airflow-postgresql-0 in StatefulSet airflow-postgresql success
23m Normal SuccessfulCreate statefulset/airflow-postgresql create Pod airflow-postgresql-0 in StatefulSet airflow-postgresql successful
3m2s Warning FailedScheduling pod/airflow-redis-0 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
23m Normal SuccessfulCreate statefulset/airflow-redis create Claim redis-db-airflow-redis-0 Pod airflow-redis-0 in StatefulSet airflow-redis success
23m Normal SuccessfulCreate statefulset/airflow-redis create Pod airflow-redis-0 in StatefulSet airflow-redis successful
23m Normal Scheduled pod/airflow-run-airflow-migrations-zrst9 Successfully assigned airflow/airflow-run-airflow-migrations-zrst9 to worker2
9m15s Normal Pulled pod/airflow-run-airflow-migrations-zrst9 Container image "apache/airflow:2.9.3" already present on machine
9m15s Normal Created pod/airflow-run-airflow-migrations-zrst9 Created container run-airflow-migrations
9m15s Normal Started pod/airflow-run-airflow-migrations-zrst9 Started container run-airflow-migrations
2m1s Warning BackOff pod/airflow-run-airflow-migrations-zrst9 Back-off restarting failed container run-airflow-migrations in pod airflow-run-airflow-migrations-zrst9_airflow(fc09b970-4e12-4e72-ba77-f7dd8acfde73)
23m Normal SuccessfulCreate job/airflow-run-airflow-migrations Created pod: airflow-run-airflow-migrations-zrst9
23m Normal Scheduled pod/airflow-scheduler-7d588dbc6-j6svn Successfully assigned airflow/airflow-scheduler-7d588dbc6-j6svn to worker2
5m26s Normal Pulled pod/airflow-scheduler-7d588dbc6-j6svn Container image "apache/airflow:2.9.3" already present on machine
9m31s Normal Created pod/airflow-scheduler-7d588dbc6-j6svn Created container wait-for-airflow-migrations
9m31s Normal Started pod/airflow-scheduler-7d588dbc6-j6svn Started container wait-for-airflow-migrations
2m32s Warning BackOff pod/airflow-scheduler-7d588dbc6-j6svn Back-off restarting failed container wait-for-airflow-migrations in pod airflow-scheduler-7d588dbc6-j6svn_airflow(d5db782d-fa0b-454f-971c-3ca91a1eb012)
23m Normal SuccessfulCreate replicaset/airflow-scheduler-7d588dbc6 Created pod: airflow-scheduler-7d588dbc6-j6svn
23m Normal ScalingReplicaSet deployment/airflow-scheduler Scaled up replica set airflow-scheduler-7d588dbc6 to 1
23m Normal Scheduled pod/airflow-statsd-6df8564664-tbsbv Successfully assigned airflow/airflow-statsd-6df8564664-tbsbv to worker2
23m Normal Pulled pod/airflow-statsd-6df8564664-tbsbv Container image "quay.io/prometheus/statsd-exporter:v0.26.1" already present on machine
23m Normal Created pod/airflow-statsd-6df8564664-tbsbv Created container statsd
23m Normal Started pod/airflow-statsd-6df8564664-tbsbv Started container statsd
23m Normal SuccessfulCreate replicaset/airflow-statsd-6df8564664 Created pod: airflow-statsd-6df8564664-tbsbv
23m Normal ScalingReplicaSet deployment/airflow-statsd Scaled up replica set airflow-statsd-6df8564664 to 1
3m2s Warning FailedScheduling pod/airflow-triggerer-0 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
23m Normal SuccessfulCreate statefulset/airflow-triggerer create Claim logs-airflow-triggerer-0 Pod airflow-triggerer-0 in StatefulSet airflow-triggerer success
23m Normal SuccessfulCreate statefulset/airflow-triggerer create Pod airflow-triggerer-0 in StatefulSet airflow-triggerer successful
23m Normal Scheduled pod/airflow-webserver-76b54cbbfc-x8t8p Successfully assigned airflow/airflow-webserver-76b54cbbfc-x8t8p to worker2
9m15s Normal Pulled pod/airflow-webserver-76b54cbbfc-x8t8p Container image "apache/airflow:2.9.3" already present on machine
9m15s Normal Created pod/airflow-webserver-76b54cbbfc-x8t8p Created container wait-for-airflow-migrations
9m15s Normal Started pod/airflow-webserver-76b54cbbfc-x8t8p Started container wait-for-airflow-migrations
2m Warning BackOff pod/airflow-webserver-76b54cbbfc-x8t8p Back-off restarting failed container wait-for-airflow-migrations in pod airflow-webserver-76b54cbbfc-x8t8p_airflow(7c1ee79d-4eee-4da8-876c-b4cdff7a0e10)
23m Normal SuccessfulCreate replicaset/airflow-webserver-76b54cbbfc Created pod: airflow-webserver-76b54cbbfc-x8t8p
23m Normal ScalingReplicaSet deployment/airflow-webserver Scaled up replica set airflow-webserver-76b54cbbfc to 1
3m2s Warning FailedScheduling pod/airflow-worker-0 0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
23m Normal SuccessfulCreate statefulset/airflow-worker create Claim logs-airflow-worker-0 Pod airflow-worker-0 in StatefulSet airflow-worker success
23m Normal SuccessfulCreate statefulset/airflow-worker create Pod airflow-worker-0 in StatefulSet airflow-worker successful
3m21s Normal FailedBinding persistentvolumeclaim/data-airflow-postgresql-0 no persistent volumes available for this claim and no storage class is set
3m21s Normal FailedBinding persistentvolumeclaim/logs-airflow-triggerer-0 no persistent volumes available for this claim and no storage class is set
3m21s Normal FailedBinding persistentvolumeclaim/logs-airflow-worker-0 no persistent volumes available for this claim and no storage class is set
3m21s Normal FailedBinding persistentvolumeclaim/redis-db-airflow-redis-0 no persistent volumes available for this claim and no storage class is set
denovonet@master:/Work/airflow-helm$ ls
airflow-secrets.yaml values.yaml
denovonet@mas

[new-php]

Setting up Apache Airflow on Ubuntu 24.04 can be a bit tricky, especially if you're new to the process. From the logs, it looks like the main issue is related to PersistentVolumeClaims (PVCs) and storage configurations in Kubernetes. This is a common hurdle when deploying stateful applications like Airflow. Here are a few steps to troubleshoot and resolve the issue:

Check StorageClass Configuration: Ensure that a StorageClass is defined in your Kubernetes cluster. If you're using a cloud provider like AWS, GCP, or Azure, they usually provide default StorageClasses. You can check this by running:
bash
Copy

kubectl get storageclass

If no StorageClass is available, you’ll need to create one or configure your cluster to use dynamic provisioning.

Manually Create PersistentVolumes: If dynamic provisioning isn’t an option, you can manually create PersistentVolumes and bind them to the PVCs. This requires pre-allocated storage (e.g., NFS, local storage, or cloud disks).

Modify values.yaml: If you’re using the default values.yaml from the Airflow Helm chart, you might need to customize it to specify the StorageClass or disable persistence for certain components if they’re not required.

Check Node Availability: Ensure that your Kubernetes nodes have enough resources (CPU, memory, and storage) to schedule the Airflow components.

For more detailed guidance on setting up Kubernetes and Helm, you might find this guide on how to Install Apache Airflow on Ubuntu 24.04 helpful. It provides a step-by-step approach to getting Airflow up and running smoothly.

Good luck with your Airflow setup! Once you get past the storage issues, it’s a powerful tool for orchestrating workflows."