Removing Phantom Container Volume in vSphere 6.7u3

Posted On // Leave a Comment
I've been playing around with Tanzu Kubernetes Grid on my vSphere 6.7u3 home lab setup, and so I've been creating and deleting clusters a lot lately. In browsing around vCenter, I noticed a new section in the Cluster and Datastore "Monitor" section called "Cloud Native Storage -> Container Volumes" that shows me all the persistent volumes that have been created with the more recent vSphere CSI Driver. Since I had been setting up my clusters with a default storage class like the following, I was now also able to visualize those volumes in the vCenter UI which is pretty neat:
kind: StorageClass
  name: vsphere-sc
  annotations: "true"
  storagepolicyname: "k8s Storage Policy"
  fstype: ext3
The new CSI based plugin ( ) allows you to simply tag various Datastores and have those automatically chosen based on the storage policy requested for the persistent volumes created in Kubernetes. Setting a default one is nice, too, since that will get used for any PVCs created that don't specify a specific storage class. That being said, I noticed that I had a persistent volume lingering in my list from a previous cluster:

In my lab, I'm using iSCSI attached disk to the cluster, those persistent volumes are being backed by standalone VMDKs separately manageable from a VM. This isn't a super new feature, as it has been around in vSphere since 6.5 (, but it is a feature that I hadn't really done too much with until working more with Kubernetes. Great, but I want to get rid of that orphaned disk!
First, I tried the absolute wrong way to delete that disk by digging around the datastore, finding an "fcd" (First Class Disk?) folder and trying to hand delete the VMDK files. Don't do this. It doesn't seem to work, and there must be some other inventory information (maybe stored in vCenter) that lingers. Only after I tried this did I find the PowerCLI "Delete-VDisk" command, which is maybe what I should have tried to delete that disk first.

Ok, so how do I fix this situation? I tried all kinds of combinations to try and fool vCenter by creating PV's manually, and then trying to point a new manual PV with the same name to an existing VDisk. No luck. I then tried to delete my cluster to see if things would get cleaned up. Now I have two entries in the container volumes list. Ugh. So then I start poking around in the vCenter SOAP API by going to front page of vCenter's Web interface, and then clicking on the "Browse Objects Managed by vSphere" link. That link takes you to a browser interface to the underlying SOAP API, and let's you snoop around in the internals. Be careful poking around because actions here can break things! I then clicked on the "content" link near the top of the page, which leads to a lot of interesting APIs you can interact with. I then clicked on the "VStorageObjectManager" link near the bottom of my list, and then on the "ListVStorageObject" API call. That call wants you to pass in a MOID (Managed Object ID) for the datastore you want to list VStorage objects from. You can get the MOID from some of the other APIs, but I cheated and used PowerCLI to use the `Get-Datastore | Select Name,Id` command to list out all my datastores and their IDs. The ID returned in this command isn't the actual MOID, but it contains the MOID you want to pass into the "ListVStorageObject" API call. You just have to strip off the leading "Datastore-" to get the MOID. So if the ID is "Datastore-datastore-12345", then the MOID would be "datastore-12345". Paste that value into the popup window that opened when we clicked the "ListVStorageObject" API link earlier, and you can probably see a list of IDs.

If you go back to the first screen we saw the phantom volume at first, and then click on the little "box" icon on the left side of that phantom entry, you will get the details view I showed above. If you click on the "Basics" tab you should see the volume ID that the container volume was attached to. This ID should match the ID you see in the list of IDs you get back with you invoke the "ListVStorageObject" API. Now, you can go back to the API listing window, and then click on the "DeleteVStorageObject_Task" API. Fill in the MOID for the datastore, and the ID of the volume we're trying to remove. If that call fails because you made the same mistake of deleting the files in the "fcd" folder, then you might have to also have to invoke another API call from the list called "ReconcileDatastoreInventory_Task". I can't be certain the proper order for all this until I test it again, and I'll try to do that if I get some time to come back and clean these tips up, but maybe all this will help future me or others.