{"id":14421,"date":"2026-10-03T23:12:09","date_gmt":"2026-10-03T21:12:09","guid":{"rendered":"https:\/\/irg-net.com\/?p=14421"},"modified":"2026-10-04T00:02:29","modified_gmt":"2026-10-03T22:02:29","slug":"vcf-9-1-%e2%86%92-9-1-1-failed-to-upload-nsx-bundle-in-the-nsx-upgrade-coordinator","status":"publish","type":"post","link":"https:\/\/irg-net.com\/index.php\/vcf-9-1-%e2%86%92-9-1-1-failed-to-upload-nsx-bundle-in-the-nsx-upgrade-coordinator\/","title":{"rendered":"VCF 9.1 \u2192 9.1.1: “Failed to upload NSX bundle” in the NSX Upgrade Coordinator"},"content":{"rendered":"
\n
\"\"<\/figure>\n<\/div>\n\n\n
<\/div>\n\n\n\n

Today I wrote a field report from a VMware Cloud Foundation 9.1 to 9.1.1 maintenance upgrade: the order that matters, the error that stopped the NSX Manager upgrade, how to prove the bundle was not the problem, and what got the upgrade moving again.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Abstract<\/h2>\n\n\n\n
    \n
  • In VCF 9.1.1 the NSX Manager upgrade is started from VCF Operations (Build \u2192 Lifecycle), not from the NSX UI \u2013 and only after the Fleet Lifecycle components and SDDC Manager are already on 9.1.1.<\/li>\n\n\n\n
  • During the NSX step the upgrade failed with \u201cNSX Upgrade Coordinator \u2013 Failed to upload NSX bundle\u201d<\/strong>. The SDDC Manager log showed an HTTP 500<\/strong> returned by one NSX Manager node on POST \/api\/v1\/upgrade\/bundles.<\/li>\n\n\n\n
  • The bundle on SDDC Manager was intact (size and SHA-256 matched), disk space was fine, and the NSX cluster was STABLE \u2013 so the problem was on the NSX Manager side.<\/li>\n\n\n\n
  • A rolling reboot of the three NSX Managers, one at a time and with a cluster-status check in between, cleared it. The retried upgrade ran through.<\/li>\n<\/ul>\n\n\n\n
    <\/div>\n\n\n\n

    Prerequisites: Everything above NSX must already be on 9.1.1<\/h3>\n\n\n\n

    Before touching NSX, be clear about where you are in the upgrade chain. VCF 9.1.1 is a maintenance release, and the patch order is dependency-driven. The NSX Manager step can only start once the management plane \u201cabove\u201d it is on 9.1.1:<\/p>\n\n\n\n

      \n
    • Fleet Lifecycle<\/strong> (the lifecycle services that drive everything else) \u2013 patched first.<\/li>\n\n\n\n
    • VCF services runtime<\/strong> and the other VCF Management components in the order the release notes dictate (VCF Operations is not patched in parallel with others; Identity Broker and Salt RaaS only after the services runtime; Migration Service Engine only after VCF Automation; patching a Software Depot blocks other patches).<\/li>\n\n\n\n
    • SDDC Manager<\/strong> \u2013 must be on 9.1.1 before any workload- or management-domain component is upgraded.<\/li>\n<\/ul>\n\n\n\n

      Only then follow the core domain order: SDDC Manager \u2192 NSX Manager \u2192 vCenter \u2192 ESX \u2192 NSX Edge\/finalize<\/strong>. Do not use \u201cUpgrade All\u201d before you have complied with these dependencies. In our environment, the Software Depot sync had run, Fleet Lifecycle and the services runtime were upgraded, and SDDC Manager was on 9.1.1:<\/p>\n\n\n\n

      <\/div>\n\n\n
      \n
      \"\"<\/figure>\n<\/div>\n\n\n
      <\/div>\n\n\n\n
      \n

      Tip<\/strong>: Right after the services runtime update, VCF Operations may show an \u201cupstream connect error or disconnect\/reset before headers\u201d message<\/strong>. In our case this was transient while the nodes rotated and it disappeared on its own after a few minutes. Don\u2019t start fixing things while the platform is still settling.<\/em><\/p>\n<\/blockquote>\n\n\n\n

      \"\"<\/figure>\n\n\n\n
      <\/div>\n\n\n\n

      Confirmation that the prerequisite is met \u2013 VCF Operations \u2192 Build \u2192 Lifecycle \u2192 VCF Instances \u2192 <instance> \u2192 <management domain> \u2192 Component Versions. SDDC Manager is \u201cOn Target\u201d (9.1.1.0); NSX, vCenter and ESX still show \u201cVersion Drift\u201d, i.e. they are the next in line:<\/p>\n\n\n\n

      <\/div>\n\n\n\n
      \"\"<\/figure>\n\n\n\n
      <\/div>\n\n\n\n

      Pre-Flight: Set up the backups BEFORE! you start – and start them again right before NSX<\/h3>\n\n\n\n

      Do not start the upgrade without working backups of both SDDC Manager and NSX.<\/strong> Both must be configured (target, credentials, schedule) before the first component is patched. A configured schedule is not enough on its own \u2013 the last backup might be hours old or, as in our SDDC Manager case, automatic backups may be disabled altogether.<\/p>\n\n\n\n

      Recommended routine \u2013 two backup runs per component:<\/strong><\/p>\n\n\n\n

        \n
      1. First backup start:<\/strong> configure the backup target and trigger a manual backup before you patch the very first component (before Fleet Lifecycle \/ SDDC Manager). This also proves that the target is reachable and the credentials work.<\/li>\n\n\n\n
      2. Second backup start:<\/strong> immediately before the NSX Manager upgrade, trigger a fresh manual backup of SDDC Manager and of NSX again, and check that both finish as \u201cSuccessful\u201d. The state you want to be able to roll back to is the one right before NSX is touched \u2013 SDDC Manager is already on 9.1.1 at that point, and the schedule may not have run since.<\/li>\n<\/ol>\n\n\n\n

        Where to do it:<\/p>\n\n\n\n

          \n
        • SDDC Manager:<\/strong> Operate \u2192 SDDC Manager \u2192 Backup Settings (SFTP target, then trigger the backup manually). In our case the last backup was successful at 12:56 PM, taken manually because automatic backup was disabled.<\/li>\n\n\n\n
        • NSX Manager:<\/strong> System \u2192 Backup & Restore. Configure the SFTP target and a schedule (every 4 hours in our case), then start a manual backup and wait for \u201cLast backup successful\u201d.<\/li>\n<\/ul>\n\n\n\n

          Also check the NSX cluster health<\/strong> \u2013 three managers, cluster STABLE, no unexplained alarms. Our only alarms were remote-logging and license-expiry related, which do not block the upgrade.<\/p>\n\n\n\n

          <\/div>\n\n\n
          \n
          \"\"
          SDDC Manager backup: last backup successful<\/sub><\/strong><\/em><\/figcaption><\/figure>\n<\/div>\n\n
          \n
          \"\"
          NSX Backup & Restore: SFTP target, every 4 hours, last backup successful.<\/sub><\/strong><\/em><\/figcaption><\/figure>\n<\/div>\n\n
          \n
          \"\"
          NSX Appliances: three managers, cluster Stable.<\/sub><\/strong><\/em><\/figcaption><\/figure>\n<\/div>\n\n\n
          <\/div>\n\n\n\n

          Where to start the NSX upgrade:<\/strong> from VCF Operations, not from the NSX UI. If you open System \u2192 Upgrade in NSX, you get this warning: \u201cThis NSX Manager is managed by VCF Operations. Upgrading NSX through this workflow may impact VCF Operations.<\/strong>\u201d Take it seriously and drive the upgrade from Build \u2192 Lifecycle \u2192 VCF Instances \u2192 <instance> \u2192 <domain> \u2192 Upgrades.<\/p>\n\n\n\n

          <\/div>\n\n\n
          \n
          \"\"<\/figure>\n<\/div>\n\n\n
          <\/div>\n\n\n\n

          The Problem<\/h3>\n\n\n\n

          The NSX upgrade plan was started from VCF Operations. Shortly after the NSX Upgrade Coordinator step began, the task failed with:<\/p>\n\n\n\n

          <\/circle><\/circle><\/circle><\/g><\/svg><\/span>