Deep Dive of the Day VPC transit gateways and external connections in VCF 9.1
If you've started using NSX VPCs in VCF 9.1, the part that usually takes longest to click is how a VPC actually reaches the outside world. Broadcom's model has three layers. A VPC is an on-demand logical network inside an NSX project. Every project, including Default, gets a transit gateway that joins its VPCs together and links them outwards. And the transit gateway attaches to one or more external connections, which are the real links into your physical network. A VPC Connectivity Profile glues it all together: it names the transit gateway, the IP blocks the VPC may use, and whether Default Outbound NAT is switched on.
There are three kinds of external connection, and your choice shapes everything else. A centralised connection goes through a Tier-0 gateway on NSX Edge nodes, which gives you BGP with the fabric and stateful services, and lets you configure NAT on the transit gateway. A distributed VLAN connection skips Edges entirely and connects the transit gateway straight to fabric VLANs on the ESX hosts, with stateful services coming from Virtual Network Appliances (VNAs). From 9.1.1 there's also a distributed VXLAN connection for EVPN fabrics, again using VNAs for advanced services. Attaching a transit gateway to a centralised connection makes it a centralised transit gateway (CTGW); attaching it to a distributed one makes it a distributed transit gateway (DTGW).
A few rules from TechDocs are easy to trip over. All the external connections on one transit gateway must be the same type, so you can't mix centralised and distributed. A VPC connects to only one transit gateway at a time, although you can move it by changing its connectivity profile, and NSX runs IPAM and span checks when you do. Public subnets and external IPs can't overlap across different DTGWs. And a CTGW can now run its own HA mode on an Edge cluster that differs from the Tier-0's: active-active suits throughput, active-standby suits NAT. That fixes the old limitation where an active-active Tier-0 meant no auto SNAT, and sharing the Tier-0's HA mode is now deprecated.
Then there's the question of which source address a packet leaves with, which Part 2 of the new VCF blog series answers neatly. Public subnets come from external IP blocks and are routed as they are. Private-VPC subnets only get out through NAT. Private-TGW subnets can talk to other VPCs on the same transit gateway without NAT, but need NAT at the CTGW or explicit advertisement to leave. Default Outbound NAT applies first (at the VPC gateway for Private-VPC, at the CTGW for Private-TGW), the CTGW routing table then picks the exit by longest prefix, and the exit connection either applies Provider Outbound SNAT, passes the address through, or forwards advertised Private-TGW traffic. If the source isn't authorised on that exit connection, there's no return path, and that's the failure the series sees most.
For production, decide early whether you need dynamic routing and centralised stateful services (centralised connections, Edges and a CTGW) or would rather keep things light and host-based (distributed connections and VNAs). Plan your IP blocks before you build anything, because overlaps between external blocks, VLANs, public and private subnets and transit gateway blocks cause failures that look like misconfiguration. Remember that stateful gateway firewalling on VPCs needs a vDefend licence. If you use VNAs, size them properly: community testing describes Small (2 vCPU, 4 GB) as lab-only and Large (8 vCPU, 32 GB) for production, with services running active/standby rather than active/active.
In a nested lab, the distributed VLAN route is usually the easiest win, since there's no Edge cluster to feed. The external VLAN has to be present on the hosts' uplinks alongside the NSX overlay vmkernel adapter, which in practice means a Layer 2 design where every host can reach it. Watch for the auto-created connectivity profile having no private transit gateway block (add one if VPCs need to talk to each other), keep the external IP block inside the distributed connection's CIDR, and add a small VNA cluster if you want Default Outbound NAT. It's also worth testing vMotion and a host failure, not just steady-state pings, before you trust the design.
Based on Transit Gateways (TechDocs, VCF 9.1) Official External Connections (TechDocs, VCF 9.1) Official Virtual Private Cloud in NSX (TechDocs, VCF 9.1) Official Tenant Connectivity on VCF 9.1, Part 2 (VCF blog, 20 Oct) Official VCF 9.1 – VNA and VPCs (SDN-Warrior, 11 May) VCF 9.1 Home Lab Series Part 14 (vStellar, 31 Jul)