4. Use-Case D One Tenant, Three Exits

This part builds the third steady-state design, which combines the previous ones. The tenant is Riyadh: workloads on all three subnet types, Public, Private-TGW and Private-VPC, showing their connectivity over three exits on one Transit Gateway: to the enterprise data centre over WAN, which routes the tenant's Private-TGW range as is (advertised, no NAT on the path); to the provider's shared services, which accept traffic only from a provider-assigned pool that is never handed to the tenant (Provider Outbound SNAT on the services connection); and to the Internet, which accepts only the public IP address.

The Requirement

Riyadh (project Tenant-Riyadh) has workloads on a public subnet, a Private-TGW subnet and a Private-VPC subnet. They need three exits, each with a different rule for the source address:

• The enterprise data centre (10.100.0.0/16), over a VRF dedicated to the tenant. The data centre must be able to use the tenant's Private-TGW addresses directly, so Private-TGW workloads must arrive there untranslated.
• The provider's shared services (10.33.0.0/24), a network the provider offers to all its tenants. The services network must see every tenant only through a provider-assigned pool, never through a tenant's own addressing.
• The Internet, through the default route, where private workloads are hidden behind the tenant's Default Outbound NAT.

Topology

Design at a glance.

Address plan.

Note: The four ranges that define the design are 20.0.0.0/16, 10.99.0.0/24, 10.100.0.0/16 and 10.33.0.0/24. The private blocks, subnet sizes and host addresses are illustrative values chosen so the packet walks can show concrete addresses. 20.0.0.x means an address NSX allocates from the block.

Steady-State Design

The resulting CTGW forwarding table. Exactly one default route, as Rule 2 requires:

10.100.0.0/16   ->  Riyadh-wan-conn        (PROVIDER_STATIC, remote network)
10.33.0.0/24    ->  Shared-services-conn   (PROVIDER_STATIC, remote network)
0.0.0.0/0       ->  Riyadh-internet-conn   (default route: no remote networks configured)

Build Order

Provider first. Create the two External IP Blocks, Riyadh-public-block_20.0.0.0/16 and Shared-services-snat-block_10.99.0.0/24; then the three connections with their advanced settings (Private-Transit Gateway IP Blocks on Riyadh-wan-conn; Provider Outbound SNAT with Shared-services-snat-block on Shared-services-conn; nothing on Riyadh-internet-conn); then the project with the three connections and Riyadh-public-block.

Riyadh-wan-conn
  Type ................................ Centralized
  Tier-0 Gateway ...................... Riyadh-wan-vrf (VRF)
  Remote Networks ..................... 10.100.0.0/16
  External IP Blocks .................. Riyadh-public-block_20.0.0.0/16
  Provider Outbound SNAT .............. Off
  Private – Transit Gateway IP Blocks . On

Shared-services-conn
  Type ................................ Centralized
  Tier-0 Gateway ...................... Shared-services-T0 (Tier-0)
  Remote Networks ..................... 10.33.0.0/24
  External IP Blocks .................. Shared-services-snat-block_10.99.0.0/24
  Provider Outbound SNAT .............. On
  SNAT IP Blocks ...................... Shared-services-snat-block_10.99.0.0/24

Riyadh-internet-conn
  Type ................................ Centralized
  Tier-0 Gateway ...................... Internet-T0 (Tier-0)
  Remote Networks ..................... (empty — implicit default route)
  External IP Blocks .................. Riyadh-public-block_20.0.0.0/16
  Provider Outbound SNAT .............. Off
  Private – Transit Gateway IP Blocks . Off

Tenant next. Attach the second and third connections to the Default Transit Gateway (Networking > Transit Gateways > ⋮ > Edit, then wait for the status to change from In Progress to Success); set the profile (Auto SNAT block Riyadh-public-block, Private-TGW block Riyadh-private-tgw-block_172.16.10.0/24, Default Outbound NAT on); then add the advertise rule of type Private-Transit Gateway for Riyadh-wan-conn under Routing and Forwarding > Advertise Rules.

Note: Shared-services-snat-block is authorized on the connection and does not need to be assigned to the project. The tenant never allocates from it. It is a provider-owned pool with one job: to be the SNAT target on Shared-services-conn and to be advertised toward Shared-services-T0 so that replies come back.

What NSX Programs Automatically

• On the VPC gateway: the Default Outbound NAT rule for the VPC, applied to Private-VPC traffic, translating to Riyadh-public-block.
• On the CTGW: the same Default Outbound NAT rule for Private-TGW traffic, translating to Riyadh-public-block; plus, scoped to Shared-services-conn only, a Provider SNAT rule translating any source outside Shared-services-snat-block to an address from Shared-services-snat-block, and a No-SNAT rule for sources already inside it. Nothing is created for the other two connections.
• Route advertisement: toward Riyadh-wan-vrf, the public prefixes from Riyadh-public-block and the Private-TGW block 172.16.10.0/24 as a tgws route; toward Shared-services-T0, prefixes from Shared-services-snat-block, which is what brings replies to the SNAT addresses back; toward Internet-T0, the public prefixes from Riyadh-public-block only.

Packet Walks: All Nine Combinations

Three subnet types and three exits give nine combinations. The matrix shows all of them; the hop tables that follow trace the two private workloads, which are the ones the settings act on.

Two product behaviours explain the matrix:

Note: By default, traffic from Private-TGW subnets is actively dropped on an External Connection whenever its source is not authorized there, as it would be if the public block were not authorized on the WAN connection. That is why the advertisement is needed on Riyadh-wan-conn: without it, Private-TGW traffic toward the DC is translated by Default Outbound NAT to 20.0.0.x or, with Default Outbound NAT off, dropped.

Note: With Default Outbound NAT, traffic from a Private-VPC subnet can reach the Transit Gateway and be provider-SNATed. That is the double translation in the Private-VPC → Shared-services-conn cell: once at the VPC gateway, once more at the CTGW.

The Private-TGW workload, Riyadh-app-01

Path A — to the enterprise DC (10.100.0.10)

Path B — to the provider services (10.33.0.10)

Path C — to the Internet (1.1.1.1)

The Private-VPC workload, Riyadh-db-01

Paths A and C — to the enterprise DC and to the Internet. The VPC gateway translates the source to 20.0.0.x in the same way on both paths. Both Riyadh-wan-conn and Riyadh-internet-conn authorize Riyadh-public-block, so the packet passes with no further translation. The DC therefore sees this workload as 20.0.0.x, not as 192.168.10.5: the advertisement on Riyadh-wan-conn covers the Private-TGW block only, and a Private-VPC address is gone before the CTGW is reached (Rule 1).

Path B — to the provider services (10.33.0.10), the double translation

The public workload, Riyadh-web-01

Its address 20.0.1.5 belongs to Riyadh-public-block, which both Riyadh-wan-conn and Riyadh-internet-conn authorize, so it reaches the DC and the Internet untranslated. Toward Shared-services-conn it is translated by Provider SNAT like everything else, because 20.0.0.0/16 is not an authorized block on that connection; the services see 10.99.0.x.

Verification

Test each workload toward each exit. The expected source at the far end is the matrix above:

# Riyadh-app-01 (172.16.10.5, Private-TGW)
ping 10.100.0.10   # DC sees 172.16.10.5    — no translation
ping 10.33.0.10    # services see 10.99.0.x — one translation (Provider SNAT, CTGW)
ping 1.1.1.1       # Internet sees 20.0.0.x — one translation (Default Outbound NAT, CTGW)

# Riyadh-db-01 (192.168.10.5, Private-VPC)
ping 10.100.0.10   # DC sees 20.0.0.x       — one translation (VPC gateway)
ping 10.33.0.10    # services see 10.99.0.x — two translations (VPC gateway, then CTGW)
ping 1.1.1.1       # Internet sees 20.0.0.x — one translation (VPC gateway)

# Riyadh-web-01 (20.0.1.5, public)
ping 10.100.0.10   # DC sees 20.0.1.5       — no translation
ping 10.33.0.10    # services see 10.99.0.x — one translation (Provider SNAT, CTGW)
ping 1.1.1.1       # Internet sees 20.0.1.5 — no translation

On the provider side, Riyadh-wan-vrf should hold 172.16.10.0/24 as a tgws route alongside the Riyadh-public-block prefixes, Shared-services-T0 should hold the Shared-services-snat-block prefixes, and Internet-T0 should hold Riyadh-public-block only.

Enterprise Nuances and Edge Cases

One public block on two connections. Authorizing Riyadh-public-block on both Riyadh-wan-conn and Riyadh-internet-conn is a deliberate simplification: the public subnet is reachable from both the DC and the Internet, and Private-VPC workloads appear to the DC under the Auto SNAT address. If the DC must see Private-VPC workloads under a dedicated range, use two blocks and Provider SNAT on Riyadh-wan-conn, as in use case A (Part 2). That means giving up the advertisement on that connection, because the two settings cannot coexist.

Provider SNAT on the services connection catches the public subnet too. The public block is not authorized on Shared-services-conn, so public-subnet traffic is translated toward the services like everything else. The services network sees this tenant, and every other tenant configured the same way, only through provider-owned pools. That is the point of the design. Any services-side policy keyed on source address should reference the provider pool 10.99.0.0/24, never tenant addressing.

The services SNAT pool never touches the project. Shared-services-snat-block is a provider-owned block; the tenant never allocates from it and it does not need to be assigned to the project. Advertising it toward Shared-services-T0 is what makes the replies return.

Subnet placement is an exit-visibility decision. The Private-TGW workload is translated exactly once on every path except the DC path, where it is not translated at all. The Private-VPC workload is translated at the VPC gateway on every path and a second time toward the services. Placing a workload on one subnet type or the other therefore decides what each exit is allowed to see, not only about scoping inside the VPC.

Attaching a third connection changes nothing on the first two. Every advanced setting is scoped to its own connection. Adding Shared-services-conn with Provider SNAT did not alter the advertisement on Riyadh-wan-conn or the Default Outbound NAT behaviour toward Riyadh-internet-conn.

The routing constraints still apply. Three connections means one default route and two connections with explicit, non-overlapping remote networks. Remote networks must be unique per connection; listing a prefix on two connections is not a supported way to get ECMP.

What Comes Next

Four designs, three source-address behaviours, one set of rules. Part 5 puts the designs side by side, states the two constraints that apply to all of them, and turns the material into operating practice: changing the Auto SNAT block on a live profile, moving a connection between Provider SNAT and advertisement, and what to verify after every change.