Cross-Region Transit Gateway routing setup

[mtb-xt]

Hi everyone,

From the network setup documentation, it's not 100% clear, how are we supposed to set up additional region.

We've added a secondary transit gateway hub, added the tgw/cross-region-hub-connector component, but then, the TGW in the second region doesn't have any routes to TGW in the primary region and vice-versa.

Is there a step we missed?

This is how our secondary region network stack looks like:

stacks/orgs/nsp/core/network/us-east-2/network.yaml

import:
  - orgs/nsp/core/network/_defaults
  - mixins/region/us-east-2
  - catalog/vpc
  - catalog/vpc-flow-logs-bucket
  - catalog/ec2-s2s-vpn
  - catalog/tgw/hub
  - catalog/tgw/spoke
#  - catalog/bastion
# disabled in catalog
#  - catalog/ec2-client-vpn

components:
  terraform:
    vpc:
      metadata:
        component: vpc
        inherits:
          - vpc/defaults
      vars:
        ipv4_primary_cidr_block: 10.0.16.0/20

    tgw/hub:
      vars:
        # These are all connections available for spokes in this region
        # Defaults environment to this region
        connections:
          # Hub for this region is always required
          - account:
              tenant: core
              stage: network
          # All stacks where a spoke will be deployed
          - account:
              tenant: platform
              stage: prod
            eks_component_names: # Add clusters here once deployed
              - eks/cluster

    tgw/spoke:
      metadata:
        inherits:
          - tgw/spoke-defaults
      vars:
        # there are no EKS clusters in core-network
        expose_eks_sg: false
        tgw_hub_tenant_name: core
        tgw_hub_environment_name: use2
        connections:
          # Hub for this region is always required        
          - account:
              tenant: core
              stage: network

          # All stacks where a spoke will be deployed
          - account:
              tenant: platform
              stage: prod
            eks_component_names:
              - eks/cluster

    # This alternate hub needs to be connected to the primary region's hub
    tgw/cross-region-hub-connector:
      vars:
        enabled: true
        primary_tgw_hub_region: ap-southeast-2

[milldr]

The tricky part with transit gateway is making sure every account has every route in both directions necessary. That means your network account will have a lot of routes. What you have there will connect the use2 region, but it does not connect your primary region to it. You need to add the connection in both directions in every single account that is a part of the route.

It's quite tedious, and I recommend using the AWS Reachability Analyzer to check if you missed any. It's almost always the back route that's missing.

---

That said, I have recently refactored all of these components for an engagement. For that use case, we needed a very complex, unique network that required many special cases. However, I think this new component setup is much easier to understand. With this new design, we have a component for each part of the network -- one for the VPC subnet routes, one for the transit gateway route table routes, one for the transit gateway attachment, and one for transit gateway itself. We don't quite have the public documentation full available for it yet, but I pushed up what we do have so far. Please feel free to take a look and let me know if you have any questions!

1. General usage docs with aws-tgw-hub: Alternate TGW Component Documentation cloudposse-terraform-components/aws-tgw-hub#16
2. aws-tgw-attachment: Initial Upstream cloudposse-terraform-components/aws-tgw-attachment#3
3. aws-vpc-routes: Initial Upstream cloudposse-terraform-components/aws-vpc-routes#3
4. aws-tgw-routes: Initial Upstream cloudposse-terraform-components/aws-tgw-routes#4

[Benbentwo]

var.tgw_connector_config is populated in this submodule by your cross-region-hub-connector components information (outputs vars etc - pulled from remote state lookup). this means that your component isn't able to fully lookup your cross-region-hub-connector.

[Benbentwo]

The components are definitely capable of setting up cross region peered TGWs. Before the above components, the tgw/hub creates the transit gateways, the tgw/cross-region-hub-connector component manages the TGW Peering. The tgw/spoke component sets up the routes for VPCs and TGWs. Thus you need a tgw/hub in each region you plan to deploy to of the network account. You need a tgw/cross-region-hub-connector for each additional region. Finally you need a spoke for ever region of every account you plan to connect.

The hard part is getting the configs just right based on what you need exactly.

One note as well is that any changes to VPCs will require a redeployment of the tgw/hub as when you deploy a tgw/hub it does a remote state lookup of the VPC component and saves information about it.

[milldr]

there is currently NO WAY to add necessary static routes to transit gateways, necessary for inter-region peering? These new components will be required for that?

No, the previous component do support everything necessary for inter-region peering, but do not support advanced customization such as centralized NAT gateways.

But, while following it - I'm now getting an error ...

As @Benbentwo pointed out, it's having trouble finding a component. I would guess that the stack in the primary region with tgw/hub is missing from the connections list.

[mtb-xt]

As @Benbentwo pointed out, it's having trouble finding a component. I would guess that the stack in the primary region with tgw/hub is missing from the connections list.

This whole issue should've been named "morons who can't read manuals" 😑 😑
What was missing for us, originally - is peered_region: true setting for alt region spokes.

Also, the guide doesn't mention this instruction:

        cross_region_hub_connector_components:
          use2:
            component: "tgw/cross-region-hub-connector"
            environment: "use2"

So, for posterity - to add another region - follow this guide, carefully - https://docs.cloudposse.com/components/library/aws/tgw/#alternate-regions
(for some reason, I didn't realize that there's a document in the top level TGW readme, I was looking in tgw/hub and tgw/hub_connector)

example of a platform spoke in an alternative region:

    tgw/spoke:
      metadata:
        inherits:
          - tgw/spoke-defaults
      vars:
        own_eks_component_names:
          - eks/cluster
        # This is a secondary region, so we have to point where our secondary hub is (and it connects to primary)
        tgw_hub_tenant_name: core
        peered_region: true # Required for alternate region spokes
        connections:
          # This stack, always included
          - account:
              tenant: platform
              stage: prod
            eks_component_names:
              - eks/cluster
          # TGW Hub, always included
          - account:
              tenant: core
              stage: network
              environment: use2
          - account:
              tenant: core
              stage: network
              environment: apse2
          - account:
              tenant: core
              stage: auto
              environment: apse2
            eks_component_names:
              - eks/cluster
        cross_region_hub_connector_components:
          use2:
            component: "tgw/cross-region-hub-connector"
            environment: "use2"

do not forget that you also need to change allowed CIDRs for EKS clusters in the new region, to reach them from a VPN connection from the primary region! This is because default catalog\eks.yaml file, in our case, had this:

        allow_ingress_from_vpc_accounts:
          - tenant: core
            stage: auto
          - tenant: core
            stage: network

but for alternative region, we had to add this to eks/cluster component:

        allow_ingress_from_vpc_accounts:
          - tenant: core
            stage: network
          - tenant: core
            stage: auto
            environment: apse2
          - tenant: core
            stage: network
            environment: apse2

[mtb-xt]

@milldr to your original point - those new components look awesome, and they give way more granularity, but, unfortunately, we don't have the capacity to try them out at the moment, sorry!

and thanks everyone for the help