VCF 9.1: Fixing an Exhausted NSX TEP Pool

While running a Host Add Workflow in my VCF 9.1 environment, the validation step “Validate IP Address Availability for Edge Overlay (TEP) IP Assignment” failed right away with the following error:

Failed to validate the provided spec with error [Not enough IP addresses available

in the IP address pool pool-name, needed 4 IP addresses but found 0]

Failed to validate the provided spec with error [Not enough IP addresses available

in the IP address pool pool-name, needed 4 IP addresses but found 0]

The message is pretty clear: the Host Add workflow needs 4 new tunnel endpoint (TEP) IP addresses, but the NSX IP address pool that provides them is completely used up. Every overlay-enabled transport node (ESX host and, depending on the design, NSX Edge) consumes one TEP IP per TEP interface – with two uplinks that is usually two IPs per host. A pool that was sized “just right” during bring-up fills up quickly once you start expanding clusters or deploying additional edges.

Broadcom describes this behavior in KB 417229 – Expanding VCF 9 cluster fails due to IP Address Pool exhaustion. In that article the symptom shows up later in the process: the cluster expansion fails with a transport node realization error, and in NSX the hosts show “Preparation Failed” with the reason:

ipAddressPool path=[/infra/ip-pools/<uuid>] is exhausted, no free IPs available for allocation.

ipAddressPool path=[/infra/ip-pools/<uuid>] is exhausted, no free IPs available for allocation.

The root cause is the same: not enough IP addresses available for the NSX tunnel endpoints. The KB’s resolution is to create a new, larger IP pool and assign it to the Transport Node Profile of the cluster. There are two catches with the KB, though:

• It is written for VCF 9.x in general, not specifically for VCF 9.1. The menu paths it references (System > Fabric > Profiles > Transport Node Profiles) and the linked 9.0 documentation do not match the new VCF 9.1 NSX UI anymore.
• It only describes one option. Creating a new pool is not always necessary – if your TEP subnet still has free addresses, simply extending the range of the existing pool is quicker and less invasive.

In VCF 9.1, the validation also kicks in earlier: instead of failing during host preparation, the workflow is stopped by a pre-check before anything is changed. That is good news – nothing is left half-configured, you just fix the pool and retry.

Before you Start: Check the Pool

In NSX Manager, go to Networking > IP Management > IP Address Pools and look at the pool named in the error. Note its subnet (CIDR), gateway and current IP range, and check which addresses in that subnet are still unused on your network.

If you want to see the actual allocations via API, use the NSX Manager API – the Policy API returns an empty list for TEP allocations:

GET https://<nsx-manager>/api/v1/pools/ip-pools/<ip-pool-UUID>/allocations

GET https://<nsx-manager>/api/v1/pools/ip-pools/<ip-pool-UUID>/allocations

Then decide which variant fits your situation:

Variant A

This is the fastest fix and the one I would try first. No new objects, no change to the Transport Node Profile – you just give the existing pool more addresses.

1. Log in to NSX Manager with admin privileges.
2. Navigate to Networking > IP Management > IP Address Pools.
3. Find the TEP pool, click the three-dot menu and select Edit.
4. In the Subnets column, click the subnet count / Set to open the Set Subnets dialog.
5. Expand the existing IP Ranges subnet and adjust the range – either widen the existing range (e.g. change the end address) or add a second range from the same CIDR. In my example the pool uses 10.181.45.30 – 10.181.45.120 in 10.181.45.0/24, so there is plenty of room left in the /24.
6. Leave CIDR, Gateway IP, DNS Servers and DNS Suffix unchanged.
7. Click Add (or Save for the subnet), then Apply, and finally Save for the pool.

Tip: Only use addresses that are really free in the TEP VLAN – NSX does not check for IPs used outside of NSX. The option “Check Overlap With Existing Pools” helps to avoid overlaps with other NSX pools. You can extend a range at any time, but you cannot shrink it below addresses that are already allocated.

Variant B

If the existing subnet is exhausted or you want to move to a bigger TEP network, create a new pool and swap it into the Transport Node Profile (TNP) – this is the approach from KB 417229, adapted to the VCF 9.1 UI.

Step 1 – Create the new IP pool

Following the VCF 9.1 documentation (Create an IP Pool for Tunnel Endpoint IP Addresses):

1. In NSX Manager, navigate to Networking > IP Address Pools and click Add IP Address Pool.
2. Enter a name and optionally a description.
3. In the Subnets column click Set, then Add Subnet > IP Ranges (or IP Block).
4. Enter the IPv4 range, CIDR, Gateway IP and optionally DNS Servers / DNS Suffix. Don’t mix IPv4 and IPv6 in a TEP pool – transport nodes don’t support mixed pools.
5. Click Add, then Apply, then Save.

Step 2 – Replace the pool in the Transport Node Profile

1. Navigate to System > Fabric > Hosts and open the Transport Node Profile tab.
2. Select the TNP of the affected cluster and open the Host Switch view.
3. Click EDIT in the blue banner (“To edit the following content, click EDIT”).
4. Expand the host switch.
5. Under IPv4 Assignment = IP Pool, change the IPv4 Pool from the old pool to the newly created one.
6. Save the host switch configuration and the profile.

NSX now pushes the updated profile to all transport nodes of the cluster. Check under System > Fabric > Hosts > Clusters that all hosts return to Success.

Important: With Variant B the existing hosts receive new TEP IPs from the new pool. Plan a maintenance window, make sure the new subnet is on the correct TEP VLAN (or routed to the other TEP networks, e.g. edge TEPs) and that the MTU is set correctly end-to-end. Afterwards verify overlay connectivity.

Retry the Host Add Workflow

Once the pool has enough free IPs, go back to the failed task in VCF and click Retry. The validation “Validate IP Address Availability for Edge Overlay (TEP) IP Assignment” should now pass and the workflow continues. An exhausted TEP pool is easy to fix, but it is also easy to avoid: size your TEP pools for future growth (at least two IPs per host plus headroom for additional hosts and edges) instead of for the day-one host count. If the subnet has room, extending the existing range (Variant A) is the quickest way out. If not, a new pool on the Transport Node Profile (Variant B) does the job – just keep in mind that the hosts get new TEP addresses.

The KB article mentioned is out of date. Some elements, such as Transport Node Profiles, are no longer located where they should be in VCF 9.1, which is why I decided to write this blog post. If you have any questions, please use the comments section below.