Unable to delete an “inaccessible” datastore (Zombie datastore)
I built a new storage array and migrated the VMs accordingly. But the server gave the following error message:
I noticed that the old datastore is still shown as “inaccessible” in vCenter:
And even via esxcli I don’t see the datastore anymore:
esxcli storage vmfs extent list
So I first choose the classic way, putting the Server in Maintenance Mode and “Reboot does Good”. But the inaccessible Datastore is still there:
Slowly I began to realize that it must be VMware vCenter, so I looked there to see what services were currently running and restarted them:
service-control --status
service-control --stop --all
service-control --start --all
And incredibly, the inaccessible datastore is still there, but without the error message “inaccessible”:
So I decided to restart the vCenter completely, after the restart I checked again the status of the services using the above command.
Well – no changes, so things are getting complicated. We need the dive into the postgre Database of the VMware vCenter to “kill” the Zombie datastore.
Please note that this process is only intended for HomeLabs and in production infrastructures, I recommend contacting VMware support.
Allright, first we jump over to the VMware vCenter SSH session again, choose “shell”. At the root@vcenter [ ~ ]# prompt type:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgresWe should now enter the psql tool
At the VCDB=# prompt you can type \d to list all tables. To check if everything is working. After I could see the tables (such as vpx_entity), I tried to find the datastore:
Make sure you substitute all names and id’s with the correct values! I will indicate all values to be replaced with a “#”.
SELECT id FROM vpx_entity WHERE name = 'esx04-raid5';
#put your datastore name between the ''!It gave me 1 result, just how we want it:
So now we know the ID of the datastore, let us find out if it is being connected to something:
SELECT * FROM vpx_ds_assignment WHERE ds_id=14335;
#use the id from the output you got!
There is one object in the inventory that is still connected to this datastore, let us check which object it is:
SELECT * FROM vpx_entity WHERE id=14336;
#use the entity_id from the output you got!
Well – vCLS – suprise suprise. The vCLS virtural machine is essentially an “appliance” or “service” VM that allows a vSphere cluster to remain functioning in the event that the vCenter Server becomes unavailable. It will maintain the health and services of that cluster. vCLS came with vSphere 7 update 1.
But it could also be different objects, so it does not matter in the main case. When the datastore can not be removed using the GUI, you might want to try removing it from the tables:
Do this on your own risk and be sure to create a backup of the VCSA before deleting or changing anything directly in the database.
DELETE FROM vpx_ds_assignment WHERE ds_id=14335;
DELETE FROM vpx_vm_ds_space WHERE ds_id=14335;
DELETE FROM vpx_datastore WHERE id=14335;
#use for this the ds_id not the entity_id!!And now reboot the VCSA appliance and verify that the Datastore is gone.
There we go and this is it from this How-To, if you have any questions, please leave a comment below. 🙂
Resources:
https://kb.vmware.com/s/article/2109887
https://www.virten.net/2020/05/connect-and-work-with-the-vcsa-embedded-vpostgres-database/
https://kb.vmware.com/s/article/2147285