Removing Phantom Container Volume in vSphere 6.7u3

I've been playing around with Tanzu Kubernetes Grid on my vSphere 6.7u3 home lab setup, and so I've been creating and deleting clusters a lot lately. In browsing around vCenter, I noticed a new section in the Cluster and Datastore "Monitor" section called "Cloud Native Storage -> Container Volumes" that shows me all the persistent volumes that have been created with the more recent vSphere CSI Driver. Since I had been setting up my clusters with a default storage class like the following, I was now also able to visualize those volumes in the vCenter UI which is pretty neat: 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsphere-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "k8s Storage Policy"
  fstype: ext3
The new CSI based plugin (https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/251 ) allows you to simply tag various Datastores and have those automatically chosen based on the storage policy requested for the persistent volumes created in Kubernetes. Setting a default one is nice, too, since that will get used for any PVCs created that don't specify a specific storage class. That being said, I noticed that I had a persistent volume lingering in my list from a previous cluster:

In my lab, I'm using iSCSI attached disk to the cluster, those persistent volumes are being backed by standalone VMDKs separately manageable from a VM. This isn't a super new feature, as it has been around in vSphere since 6.5 (https://www.virtuallyghetto.com/2016/11/new-vsphere-6-5-apis-worth-checking-out.html), but it is a feature that I hadn't really done too much with until working more with Kubernetes. Great, but I want to get rid of that orphaned disk!
First, I tried the absolute wrong way to delete that disk by digging around the datastore, finding an "fcd" (First Class Disk?) folder and trying to hand delete the VMDK files. Don't do this. It doesn't seem to work, and there must be some other inventory information (maybe stored in vCenter) that lingers. Only after I tried this did I find the PowerCLI "Delete-VDisk" command, which is maybe what I should have tried to delete that disk first.


Ok, so how do I fix this situation? I tried all kinds of combinations to try and fool vCenter by creating PV's manually, and then trying to point a new manual PV with the same name to an existing VDisk. No luck. I then tried to delete my cluster to see if things would get cleaned up. Now I have two entries in the container volumes list. Ugh. So then I start poking around in the vCenter SOAP API by going to front page of vCenter's Web interface, and then clicking on the "Browse Objects Managed by vSphere" link. That link takes you to a browser interface to the underlying SOAP API, and let's you snoop around in the internals. Be careful poking around because actions here can break things! I then clicked on the "content" link near the top of the page, which leads to a lot of interesting APIs you can interact with. I then clicked on the "VStorageObjectManager" link near the bottom of my list, and then on the "ListVStorageObject" API call. That call wants you to pass in a MOID (Managed Object ID) for the datastore you want to list VStorage objects from. You can get the MOID from some of the other APIs, but I cheated and used PowerCLI to use the `Get-Datastore | Select Name,Id` command to list out all my datastores and their IDs. The ID returned in this command isn't the actual MOID, but it contains the MOID you want to pass into the "ListVStorageObject" API call. You just have to strip off the leading "Datastore-" to get the MOID. So if the ID is "Datastore-datastore-12345", then the MOID would be "datastore-12345". Paste that value into the popup window that opened when we clicked the "ListVStorageObject" API link earlier, and you can probably see a list of IDs.

If you go back to the first screen we saw the phantom volume at first, and then click on the little "box" icon on the left side of that phantom entry, you will get the details view I showed above. If you click on the "Basics" tab you should see the volume ID that the container volume was attached to. This ID should match the ID you see in the list of IDs you get back with you invoke the "ListVStorageObject" API. Now, you can go back to the API listing window, and then click on the "DeleteVStorageObject_Task" API. Fill in the MOID for the datastore, and the ID of the volume we're trying to remove. If that call fails because you made the same mistake of deleting the files in the "fcd" folder, then you might have to also have to invoke another API call from the list called "ReconcileDatastoreInventory_Task". I can't be certain the proper order for all this until I test it again, and I'll try to do that if I get some time to come back and clean these tips up, but maybe all this will help future me or others.

Comments

Post a Comment

Popular posts from this blog

Using Snapshot Isolation with SQL Server and Hibernate

In the course of working on some deadlock issues, I found that MS SQL Server exhibits some unexpected (to me, at least) locking behaviors that can affect performance and cause deadlocks. In the end, I found that SQL Server provides an isolation level called Snapshot isolation which removes the need for locks on rows and removes lock contention. To use this isolation level, you need to execute the following SQL on the database that you want to use the isolation level with (replacing MyDatabase with your database name, of course): ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON Next, you need to modify your hibernate.cfg.xml file to add a property to tell Hibernate to use the Snapshot Isolation level: <!-- The 4096 isolation level is the setting to use with the jTDS or Microsoft JDBC drivers --> < property name =”hibernate.connection.isolation”>4096</property> Snapshot isolation is not a panacea...
Read more

FreeNAS, CIFS, and user login problems

Image
I have been in the process of converting from the dead-end Windows Home Server product to a FreeNAS server for my home NAS.  I was trying to replicate the same sort of set up that WHS had for user directories.  In WHS, there was a share called "Users" and under that share there were various user directories corresponding to each user account.  I thought it would be pretty simple to replicate that same behavior to allow users to simply attach to \\nas-server-name\Users in Windows, and then navigate to their own user folder.  What follows is how I achieved this, as well as a problem I fought with for a while that prevented certain users from connecting via CIFS. First, I started off the process by installing FreeNAS to a USB stick, added a few disks in the server, plugged in the USB stick and booted up.  I performed basic config for the networking, and admin account passwords, and turned on the CIFS service. Next, I created a some user accounts fo...
Read more

Official Pandora Desktop and Hotkey Support?

EDIT: There is a slight problem occasionally with this script not being able to send keys to the Pandora Desktop Air app. I've got an updated version of the script that you can use instead of this one. Pandora is a different sort of online music service that uses the work of the Music Genome Project to automatically selects music similar to an artist or particular song, and then streams it over the Internet for your listening pleasure. You can't select individual songs, but you can skip ones you don't like and make as many music channels as you like to suit your particular tastes. Pandora provides a free version of the product that is supported by visual advertising, and a very inexpensive ($36 USD per year as of the creation of this entry) subscription supported version of the product that provides higher quality streams and no advertising. Traditionally, Pandora has been a Flash-based application that required you to have your bulky browser open to listen on the deskto...
Read more