Recovering from vCenter Appliance Disk Errors on LVM Devices

Let's say you have a ghetto vSphere home lab.

And let's say that you are running a vCenter Appliance to manage that home lab because you didn't want to devote a whole physical machine to that task because you are cheap.

And let's say you are running a small storage server for that home lab that is hosting the disks for that vCenter Appliance.

And let's say that that home storage server is running on a UPS, but _sometimes_ the power goes out for a little bit longer than your UPS can handle and you haven't had the time to configure that file server to shutdown the vSphere hosts before it shuts itself down.

Everything comes back up after the power failure, but your vCenter Appliance VM is complaining about file system errors and won't boot.  How do you fix that?

Well, the good news is that there are some great guides out there to get you part of the way to a solution.  I followed http://www.opvizor.com/blog/vmware-vcenter-server-appliance-vcsa-filesystem-is-damaged/ to get out to a BASH prompt, but the filesystems that I was getting errors for were on LVM volume groups.  And when I went to look for those devices, they weren't showing up under /dev/mapper.

The problem was that those LVM volume groups were not being marked active when I booted up using the method in the procedure above.  Luckily, the commands below allow you to make sure the device nodes get created under /dev/mapper, and then you can run fsck against the failing LVM devices.

(none):/ # modprobe dm_mod
(none):/ # vgscan
  Failed to find sysfs mount point
  Reading all physical volumes.  This may take a while...
  Found volume group "invsvc_vg" using metadata type lvm2
  Found volume group "autodeploy_vg" using metadata type lvm2
  Found volume group "netdump_vg" using metadata type lvm2
  Found volume group "seat_vg" using metadata type lvm2
  Found volume group "dblog_vg" using metadata type lvm2
  Found volume group "db_vg" using metadata type lvm2
  Found volume group "log_vg" using metadata type lvm2
  Found volume group "core_vg" using metadata type lvm2
  Found volume group "invsvc_vg" using metadata type lvm2
(none):/ # vgchange -ay
  Failed to find sysfs mount point
  1 logical volume(s) found in volume group "invsvc_vg" now active
  1 logical volume(s) found in volume group "autodeploy_vg" now active
  1 logical volume(s) found in volume group "netdump_vg" now active
  1 logical volume(s) found in volume group "seat_vg" now active
  1 logical volume(s) found in volume group "dblog_vg" now active
  1 logical volume(s) found in volume group "db_vg" now active
  1 logical volume(s) found in volume group "log_vg" now active
  1 logical volume(s) found in volume group "core_vg" now active
  1 logical volume(s) found in volume group "invsvc_vg" now active
(none):/ # fsck /dev/mapper/log_vg-log
fsck from util-linux 2.19.1
e2fsck 1.41.9 (22-Aug-2009)
...

Comments

Post a Comment

Popular posts from this blog

Using Snapshot Isolation with SQL Server and Hibernate

In the course of working on some deadlock issues, I found that MS SQL Server exhibits some unexpected (to me, at least) locking behaviors that can affect performance and cause deadlocks. In the end, I found that SQL Server provides an isolation level called Snapshot isolation which removes the need for locks on rows and removes lock contention. To use this isolation level, you need to execute the following SQL on the database that you want to use the isolation level with (replacing MyDatabase with your database name, of course): ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON Next, you need to modify your hibernate.cfg.xml file to add a property to tell Hibernate to use the Snapshot Isolation level: <!-- The 4096 isolation level is the setting to use with the jTDS or Microsoft JDBC drivers --> < property name =”hibernate.connection.isolation”>4096</property> Snapshot isolation is not a panacea...
Read more

FreeNAS, CIFS, and user login problems

Image
I have been in the process of converting from the dead-end Windows Home Server product to a FreeNAS server for my home NAS.  I was trying to replicate the same sort of set up that WHS had for user directories.  In WHS, there was a share called "Users" and under that share there were various user directories corresponding to each user account.  I thought it would be pretty simple to replicate that same behavior to allow users to simply attach to \\nas-server-name\Users in Windows, and then navigate to their own user folder.  What follows is how I achieved this, as well as a problem I fought with for a while that prevented certain users from connecting via CIFS. First, I started off the process by installing FreeNAS to a USB stick, added a few disks in the server, plugged in the USB stick and booted up.  I performed basic config for the networking, and admin account passwords, and turned on the CIFS service. Next, I created a some user accounts fo...
Read more

Official Pandora Desktop and Hotkey Support?

EDIT: There is a slight problem occasionally with this script not being able to send keys to the Pandora Desktop Air app. I've got an updated version of the script that you can use instead of this one. Pandora is a different sort of online music service that uses the work of the Music Genome Project to automatically selects music similar to an artist or particular song, and then streams it over the Internet for your listening pleasure. You can't select individual songs, but you can skip ones you don't like and make as many music channels as you like to suit your particular tastes. Pandora provides a free version of the product that is supported by visual advertising, and a very inexpensive ($36 USD per year as of the creation of this entry) subscription supported version of the product that provides higher quality streams and no advertising. Traditionally, Pandora has been a Flash-based application that required you to have your bulky browser open to listen on the deskto...
Read more