How to scale beyond 2 data products / integrations

[renepajta]

Hi all,

data landing zone comes with predefined setup for 2 data products / integrations. I see the benefit for integration as mostly you will need batch and streaming part, however, there is potential for many data products.

Currently a resource group and subnet are mentioned as prerequisites for a data product / integration.

What would a be a great spot in IaC to add new RGs and subnets for addition data products? DLZ template?

Thinking about self-service, creation of RG and subnet would need to be as well automated? Would it be still a good fit for DLZ template?

[marvinbuss]

That is definitely true. We are creating these data product and data integration resources (resource groups, subnets, storage account containers) as samples to more easily get started with the development of data products and data integrations and to also make clear where to connect and how to separate teams on a data, network and RBAC layer.

Most certainly, companies will have more than two data product and data integration teams working inside a single data landing zone over time. This is also what the architecture was designed for. Data Products and Data Integrations are units of scale besides the option to scale with Data Landing Zones and subscriptions.

If more subnets, resource groups and storage account containers are required to deploy additional data products and data integrations, we are expecting the customer to do this via automation workflows. If a new team needs to be onboarded, a process needs to create the resource group, the subnet, the storage account container and make the necessary role assignments. This is often combined with a request and approval workflow, where data product or data integration teams can request a new environment. We will add some of these automation steps and scripts over the next few weeks.

We are not including the creation of subnets, resource groups and storage account containers as part of the Data Product deployment as these teams usually do not have the permissions to create subnets, storage account containers or additional resource groups. This is something that is owned by the Data Platform Ops or Data Landing Zone Ops team.