Understanding Workspaces

[adammrozik]

Hi!

I am relatively new to Dagster and I have to say that I love how it looks like! We have been looking lately at using Dagster as an orchestration tool for our pipelines and there is one thing that we are trying to solve. Hopefully you can help me with answering some questions:

Our goal is to run Dagster on Kubernetes. We will be having multiple projects running, each with its own set of python dependencies.
I understand that Dagster workers have to have all of the packages available, but if we have the same workers for all of the projects, it is hard to do.

My question would be: How to have workers use different environments depending on what code is being run? I started reading into Workspace/GRPC, but I am not certain if I understand how it works correctly.

Right now we are simply pushing new docker image for every single commit in the repository, but obviously we cannot have different workers for each of the pipelines.

My current solution resolves around having multiple environments inside the worker and have the projects designed so that they create a pipeline that uses only one of those predefined environment.

However, from the example in here (https://docs.dagster.io/deploying/kubernetes) and here (https://dev.to/michielhub/dagster-with-user-code-deployments-grpc-2c16) it seems as though each pipeline has its own image that it will run ? Will these images replaces workers, or run on top of them? What if at the same time someone else starts its pipeline on the same Kubernetes cluster? Will there be new workers?

Would be very grateful for any Simple guidance, as I may be lacking some key understanding.

[alangenfeld]

I will answer this question in the context of our current K8sRunLauncher and how its set up in our helm chart. Run launching is a pluggable part of the system though so different behaviors can be achieved.

it seems as though each pipeline has its own image that it will run ?

With our helm setup you will have some number of "user code deployments." These have repositories which have pipelines, and dagit talks to these to represent whats available to execute. These deployments know what image they are running, and so when we launch a pipeline, we will create a Kubernetes job using that image to execute the run. The executor that is selected will determine how the run executes each solid / step.

https://kubernetes.io/docs/concepts/workloads/controllers/job/

Will these images replaces workers, or run on top of them?

The "run workers" are K8s jobs and will not be modified after creation.

What if at the same time someone else starts its pipeline on the same Kubernetes cluster? Will there be new workers?

Yep a new run means a new job pod for that run will be spun up.

[adammrozik]

Thanks for the answer! So more or less this works as follows: * I create an image with dagster installed and my repositories copied. * I deploy dagster with helm using my new image * This image, thanks to repositories section, is able to have my pipelines and run each of them * Here it gets blurry for me. Each pipeline requires it's own section "run_launcher" in dagster.yaml file to know in which image it will be executed ? Is it a different image than the one used during helm Deployment? * Also, if everything is deployed, how can I add a new pipeline? I have to redeploy helm cluster one more time? pt., 26 lut 2021, 17:51 użytkownik Alex Langenfeld <notifications@github.com> napisał:

…

I will answer this question in the context of our current K8sRunLauncher and how its set up in our helm chart. Run launching is a pluggable part of the system though so different behaviors can be achieved. it seems as though each pipeline has its own image that it will run ? With our helm setup you will have some number of "user code deployments." These have repositories which have pipelines, and dagit talks to these to represent whats available to execute. These deployments know what image they are running, and so when we launch a pipeline, we will create a Kubernetes job using that image to execute the run. The executor that is selected will determine how the run executes each solid / step. https://kubernetes.io/docs/concepts/workloads/controllers/job/ Will these images replaces workers, or run on top of them? The "run workers" are K8s jobs and will not be modified after creation. What if at the same time someone else starts its pipeline on the same Kubernetes cluster? Will there be new workers? Yep a new run means a new job pod for that run will be spun up. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3742 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARPF4NQGD3O6UAZINJ2XMZDTA7GSHANCNFSM4YIXKPGQ> .

[adammrozik]

Also, one more question: assuming each pipeline is being run as a separate Kubernetes Job, how does parallelism of solids work? When you deploy to dask for example, I guess independent solids would get executed by different workers in parallel. Would the same be true inside a Kubernetes Job? pt., 26 lut 2021, 17:51 użytkownik Alex Langenfeld <notifications@github.com> napisał:

…

I will answer this question in the context of our current K8sRunLauncher and how its set up in our helm chart. Run launching is a pluggable part of the system though so different behaviors can be achieved. it seems as though each pipeline has its own image that it will run ? With our helm setup you will have some number of "user code deployments." These have repositories which have pipelines, and dagit talks to these to represent whats available to execute. These deployments know what image they are running, and so when we launch a pipeline, we will create a Kubernetes job using that image to execute the run. The executor that is selected will determine how the run executes each solid / step. https://kubernetes.io/docs/concepts/workloads/controllers/job/ Will these images replaces workers, or run on top of them? The "run workers" are K8s jobs and will not be modified after creation. What if at the same time someone else starts its pipeline on the same Kubernetes cluster? Will there be new workers? Yep a new run means a new job pod for that run will be spun up. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#3742 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ARPF4NQGD3O6UAZINJ2XMZDTA7GSHANCNFSM4YIXKPGQ> .