Vineyard for Airflow deployed on k8s

[ravi-k4]

Is there any documentation about how to setup Vineyard for Airflow deployed on k8s?

[sighingnow]

Is there any documentation about how to setup Vineyard for Airflow deployed on k8s?

Thanks for your interest! I could see how to integration vineyard into the airflow's k8s helm deployment and provides a documentation this week.

Are your deploying airflow using helm charts from https://airflow.apache.org/docs/helm-chart/stable/index.html?

[ravi-k4]

Thanks, @sighingnow. Yes, I am using the apache airflow chart.

[sighingnow]

Hi @ravi-k4,

We have just cut a release with a reference configuration about deploying on Kubernetes with vineyard support: https://github.com/v6d-io/v6d/tree/main/python/vineyard/contrib/airflow#deploy-on-kubernetes.

The values.yaml contains information about how to add a vineyard container into the airflow pods.

Note that the values.yaml doesn't include configurations like postgresql and other airflow specific configurations.

Hope that could be helpful! If you still encounter vineyard related troubles, feel free to let me know.

[ravi-k4]

Thanks Tao! I will test it in my airflow environment. Regarding etcd - does it requires to have storage backend, will the running DAG's have an issue if etcd pod restarts?

[sighingnow]

The etcd.yaml given in the repo is just for testing propose. To deploy a etcd cluster with HA, I think things like https://github.com/bitnami/charts/tree/master/bitnami/etcd should be worth a try.