Removing Phantom Container Volume in vSphere 6.7u3

I've been playing around with Tanzu Kubernetes Grid on my vSphere 6.7u3 home lab setup, and so I've been creating and deleting clusters a lot lately. In browsing around vCenter, I noticed a new section in the Cluster and Datastore "Monitor" section called "Cloud Native Storage -> Container Volumes" that shows me all the persistent volumes that have been created with the more recent vSphere CSI Driver. Since I had been setting up my clusters with a default storage class like the following, I was now also able to visualize those volumes in the vCenter UI which is pretty neat: 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: vsphere-sc
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.vsphere.vmware.com
parameters:
  storagepolicyname: "k8s Storage Policy"
  fstype: ext3
The new CSI based plugin (https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/251 ) allows you to simply tag various Datastores and have those automatically chosen based on the storage policy requested for the persistent volumes created in Kubernetes. Setting a default one is nice, too, since that will get used for any PVCs created that don't specify a specific storage class. That being said, I noticed that I had a persistent volume lingering in my list from a previous cluster:

In my lab, I'm using iSCSI attached disk to the cluster, those persistent volumes are being backed by standalone VMDKs separately manageable from a VM. This isn't a super new feature, as it has been around in vSphere since 6.5 (https://www.virtuallyghetto.com/2016/11/new-vsphere-6-5-apis-worth-checking-out.html), but it is a feature that I hadn't really done too much with until working more with Kubernetes. Great, but I want to get rid of that orphaned disk!
First, I tried the absolute wrong way to delete that disk by digging around the datastore, finding an "fcd" (First Class Disk?) folder and trying to hand delete the VMDK files. Don't do this. It doesn't seem to work, and there must be some other inventory information (maybe stored in vCenter) that lingers. Only after I tried this did I find the PowerCLI "Delete-VDisk" command, which is maybe what I should have tried to delete that disk first.


Ok, so how do I fix this situation? I tried all kinds of combinations to try and fool vCenter by creating PV's manually, and then trying to point a new manual PV with the same name to an existing VDisk. No luck. I then tried to delete my cluster to see if things would get cleaned up. Now I have two entries in the container volumes list. Ugh. So then I start poking around in the vCenter SOAP API by going to front page of vCenter's Web interface, and then clicking on the "Browse Objects Managed by vSphere" link. That link takes you to a browser interface to the underlying SOAP API, and let's you snoop around in the internals. Be careful poking around because actions here can break things! I then clicked on the "content" link near the top of the page, which leads to a lot of interesting APIs you can interact with. I then clicked on the "VStorageObjectManager" link near the bottom of my list, and then on the "ListVStorageObject" API call. That call wants you to pass in a MOID (Managed Object ID) for the datastore you want to list VStorage objects from. You can get the MOID from some of the other APIs, but I cheated and used PowerCLI to use the `Get-Datastore | Select Name,Id` command to list out all my datastores and their IDs. The ID returned in this command isn't the actual MOID, but it contains the MOID you want to pass into the "ListVStorageObject" API call. You just have to strip off the leading "Datastore-" to get the MOID. So if the ID is "Datastore-datastore-12345", then the MOID would be "datastore-12345". Paste that value into the popup window that opened when we clicked the "ListVStorageObject" API link earlier, and you can probably see a list of IDs.

If you go back to the first screen we saw the phantom volume at first, and then click on the little "box" icon on the left side of that phantom entry, you will get the details view I showed above. If you click on the "Basics" tab you should see the volume ID that the container volume was attached to. This ID should match the ID you see in the list of IDs you get back with you invoke the "ListVStorageObject" API. Now, you can go back to the API listing window, and then click on the "DeleteVStorageObject_Task" API. Fill in the MOID for the datastore, and the ID of the volume we're trying to remove. If that call fails because you made the same mistake of deleting the files in the "fcd" folder, then you might have to also have to invoke another API call from the list called "ReconcileDatastoreInventory_Task". I can't be certain the proper order for all this until I test it again, and I'll try to do that if I get some time to come back and clean these tips up, but maybe all this will help future me or others.

Comments

Post a Comment

Popular posts from this blog

Ghetto Cloud Foundry Home Lab

Image
Over the past few months, I've been cobbling together my own Lab to be able to gain experience with Cloud Foundry.  Sure, I could have gone the much simpler route of bosh-lite , but I wanted to get a broader set of experience with the underlying IaaS layer in conjunction with working with Cloud Foundry. My lab hardware was purchased from various places (eBay, Fry's, etc) when I could get deals on it. Rocking the Ghetto Lab At a high level, the hardware looks like this: Machine CPU Memory Storage Notes HP Proliant ML350 G5 2x Intel Xeon CPU E5420 @ 2.50GHz 32 GB (Came with some disks, but mostly unused) vSphere Host 1, Added a 4 port Intel 82571EB Network Adapter HP Proliant ML350 G5 2x Intel Xeon CPU E5420 @ 2.50GHz 32 GB (Came with some disks, but mostly unused) vSphere Host 2, Added a 4 port Intel 82571EB Network Adapter Whitebox FreeNAS server Intel Celeron CPU G1610 @ 2.60GHz 16 GB 3x 240GB MLC SSDs, in a ZFS stripe set, plus spinning d
Read more

Using Snapshot Isolation with SQL Server and Hibernate

In the course of working on some deadlock issues, I found that MS SQL Server exhibits some unexpected (to me, at least) locking behaviors that can affect performance and cause deadlocks. In the end, I found that SQL Server provides an isolation level called Snapshot isolation which removes the need for locks on rows and removes lock contention. To use this isolation level, you need to execute the following SQL on the database that you want to use the isolation level with (replacing MyDatabase with your database name, of course): ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON Next, you need to modify your hibernate.cfg.xml file to add a property to tell Hibernate to use the Snapshot Isolation level: <!-- The 4096 isolation level is the setting to use with the jTDS or Microsoft JDBC drivers --> < property name =”hibernate.connection.isolation”>4096</property> Snapshot isolation is not a panacea
Read more

Fedora, Ant, and Optional Tasks

I've tried to be good. I've tried a number of times to use the version of ant that Fedora packages up into an RPM from the JPackage repos rather than caving to the temptation to install it myself manually. I also like to use optional tasks like ReplaceRegExp in my scripts to do all sorts of cool things. Of course, I see most of the installation instructions that refer to ant via RPM mention that I need to yum install ant-apache-regexp to get my optional task jars installed for this very nice task. Every time I've tried this, ant yells at me saying that it can't find the taskdef class for ReplaceRegExp. Huh? But I installed the proper, RPMs...didn't I? Usually at this point, I fall back to what works and just install ant by hand. However, I've finally realized why I've had so much trouble with this. There is one additional RPM that should be installed to get the classes that define all the optional tasks for ant called ant-nodeps. Before getting int
Read more