How to manage multitenancy onboarding within stages?

[ksawerykarwacki]

I am struggling a bit with pipeline structure.

In my case I have multitenant app deployed for each stage (dev, qa, prod, etc.) however on top of that I onboard tenants and their specific objects (ingresses, crossplane compositions, etc.). What is the most effective way to do that if I want to store everything in git?

Should I use separate stage for each tenant? I obviously do not want to run full path (dev-qa-prod) for each new tenant onboarded on prod as I do not effectively promote anything. Effectively I have three sources of change:

• artifacts releases (images, charts, crossplane packages)
• stage specific config
• tenants deployed on specific stage (which is subset of stage specific config but will change more often and do not require full promotion cycle)

Ideally, I would like to avoid doing that manually. I am thinking about having something like control flow stage pattern (https://main.docs.kargo.io/user-guide/patterns/#control-flow-stages) when I will have manually updated stages that are gatekeepers and represent stage (dev, qa, prod) and then auto updated stages that represent specific tenants within the stage. Still, it does not answer how to not run whole pipeline if I only add tenant on prod. Do I have to run separate pipeline (or even project) for each stage to avoid triggering whole promotion train?