Creating a base CentOS 6.4 Image for Pivotal HD Installation

Pivotal HD 2.0 is simple to install once you have a proper base to install to. However, there are some prerequisites and some things you can do to make the install process simpler. Although these instructions were mainly for a virtual machine based install, they can be used for a physical machine based install by just ignoring certain steps.

Virtual Hardware

I started off with a small VM to build the image on. I can always re-size the virtual hardware later on if needed. The base VM I started with was 1 CPU, 2 GB Memory and 8 GB of disk. If you are installing to physical, you probably need to follow the recommended sizing guide for Pivotal HD at

Installing CentOS

Next, I installed CentOS 6.4 using the minimal ISO which can be downloaded from Either burn it to a CD, or USB stick (, or use your virtualization software to attach the ISO to the VM and boot from it.
Boot menu after booting from CentOS 6.4 Minimal ISO
After selecting the first option in the boot menu, you'll be asked if you want to test the install media. Skip it unless you want to wait an long time, and you are unsure the install ISO is valid. The install will switch into graphics mode. Just select whatever makes sense for your install from the language and keyboard dialogs. Next, select "Basic Storage Devices" when asked "What types of devices will your installation have?" You will probably see a dialog pop up that is warning you that your disk may contain data and you are likely to lose it. If this is a new VM with a new disk, then you are fine to click "Yes, discard my data". If this is a physical server, double check that you have the correct disk shown in the dialog, and click "Yes" only if you are sure it is ok to wipe out. If not, select "No, keep any data" and move on. You usually only would use "No" if you have another operating system on the same disk that you want to retain. Next, give your machine a host name. I set mine to centos6base.local, but you are free to use whatever you like here. If you are installing this machine as a virtual machine that you will clone, use some host name you are likely not to use for your cluster.  The most important thing to do here is to click the "Configure Network" button.
Configuring the Host Name
In the resulting dialog, select the "System eth0" adapter, and click the "Edit" button.  Check the "Connect Automatically" check-box, and click "Apply" then "Close" and then "Next" to continue.
Make sure to connect automatically to the network
You could certainly configure a static IP address here if you know you are already going to need it, but leaving the adapter to configure itself via DHCP is a reasonable starting point if you are on a trusted network.  If this is a Virtual Machine image you are building, then it may also make sense to start with DHCP for the time being, and then change the machine later on to use a static IP if needed.  If you set a static address, make sure to set the proper DNS server entries to allow name resolution to work.

Next you need to select your timezone, and enter a root password (don't forget it!).

After selecting your root password and clicking next, you'll be presented with a screen to customize your disk partitioning.  Choose "Use all space", check the "Review and modify partitioning layout" check-box, and click "Next".  In the resulting dialog, make sure that the "lv_swap" isn't more than 2048, and re-size it if it is so that you can use the rest of the space for lv_root.  When you are satisfied with the the partitioning, click "Next" to continue.
Reviewing partitioning
You will likely be presented with warning that the disk is going to be formatted and all data removed.  This is your last chance to save anything you might have on that disk that you want to keep.  If you are good with wiping out that disk, click "Format", and then "Write changes to disk".

Click "Next" in the next dialog to just install the GRUB boot loader with it's default settings.  Now you can grab a quick coffee and wait for the install to finish.  Remove the install media from the machine, and click the "Reboot" button.

Further Customization

When the machine reboots and you log back in as root, whatever you do, do _not_ call yum update! This will upgrade your image to a later version of CentOS which will not work with Pivotal HD.  We do want to install a few more packages, though, using the following command:

yum -y install perl wget ntp ed samba-client cifs-utils

The samba-client and cifs-utils packages aren't required, but they are nice to have if you are in a Windows environment and need to mount shares.


Windows Shares

If you are using Windows shares, and you want to be able to resolve Windows machine names, you'll need to edit /etc/nsswitch.conf to add WINS name resolution.  You can call the following to change your nsswitch.conf file:

sed -i "s/^hosts:.*/hosts:  files wins dns/" /etc/nsswitch.conf


Pivotal HD requires that SELinux be set to permissive or disabled.  To disable SELinux, issue the following command:

sed -i 's/SELINUX=[a-z]*/SELINUX=disabled/' /etc/selinux/config

Disable iptables and ip6tables

Pivotal HD currently requires that the local firewall be disabled as well.  You can disable the firewall with the following commands:

chkconfig iptables off
chkconfig ip6tables off
service iptables stop
service ip6tables stop

Enable NTP

Time synchronization is very important for a stable cluster.  To ensure consistent times across the cluster, we'll use NTP.  You can enable NTP with the following commands:

chkconfig ntpd on
service ntpd start

Multicast DNS with avahi

Setting up a DNS server just for a Pivotal HD cluster can be a pain.  Also, even if you do have a DNS server, you may not have the ability to add new entries to it.  So, if you'd rather avoid having to setup or modify DNS for your cluster, you can use multicast DNS with a service called avahi, and a corresponding name resolution model called mdns.

Multicast DNS allows you to resolve hosts using a ".local" domain.  So if you set your machine's host name to foo, then you could ping "foo.local" from another machine that has mDNS name resolution enabled.
For you Mac users out there, multicast DNS is how Mac Bonjour works.

The avahi service relies on another service called D-Bus, which isn't in the default CentOS repos.  To get D-Bus, you'll have to add another yum repo to your machine.  The blog post at provides instructions that are close to what you need to do to add the ATRPMs repo, but below are the steps I had to follow to get it to work:

rpm --import
rpm -ivh
sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/atrpms.repo

Next, we install the avahi and nss-mdns packages:

yum -y --enablerepo=atrpms install nss-mdns avahi

Finally, we enable the avahi service, and start D-Bus and avahi up.

chkconfig avahi-daemon on
service messagebus start
service avahi-daemon start

The above instructions enabled this machine to respond to mDNS requests, but we also want to be able to resolve names on this machine via mDNS.  To do that, we need to modify the /etc/nsswitch.conf file.  Remove the "wins" part of the command below if you don't need name resolution using WINS.

sed -i "s/^hosts:.*/hosts:  files mds4_minimal [NOTFOUND=return] wins dns mdns4/" /etc/nsswitch.conf

Enable password-less SSH between cluster hosts

To allow Pivotal HD to install, you will need to enable password-less SSH between the cluster hosts.  First you need to generate keys with the following command:


Next, if you are building a virtual machine base template, you can pre-share the key you just generated with the following commands:

cat ~/.ssh/ >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

Now, when you clone the virtual machine, you will have the same keys for root and the same public key in  authorized_keys for all your machines.  If you are installing multiple machines by hand, then you can use the ssh-copy-id script to copy your key to each of the machines in your cluster by hand (replace or set $A_MACHINE with the IP or host name of the host you want to copy your public key to):

ssh-copy-id root@$A_MACHINE

Install Oracle 1.7 JDK RPM

Pivotal HD requires Oracle JDK 1.7 to be installed.  You will need to download it using a browser, as Oracle requires you to accept their licensing agreement before downloading.  You can download the 1.7 JDK from links on  Download the 64 bit RPM to make installation easier.  After downloading, you can scp (or perhaps pscp for Windows) the file to your new machine.

After getting the file up to your machine, you can install the JDK with the following command in the directory you uploaded the RPM to:

yum -y localinstall jdk-7u*-linux-x64.rpm

Update alternatives

Unfortunately, Oracle's RPM doesn't properly update the alternatives links on CentOS.  To update those links, issue the following command:

alternatives --install /usr/bin/java java /usr/java/default/bin/java 1 \
--slave /usr/bin/jar jar /usr/java/default/bin/jar \
--slave /usr/bin/javac javac /usr/java/default/bin/javac \
--slave /usr/bin/javadoc javadoc /usr/java/default/bin/javadoc \
--slave /usr/bin/javaws javaws /usr/java/default/bin/javaws \
--slave /usr/bin/jcontrol jcontrol /usr/java/default/bin/jcontrol

Additional Steps for Virtual Machine Template

If you are creating a base template to use for Virtual based installs, then you will need to perform the following steps.

Install VMware Tools

In vSphere (or Fusion or Workstation), execute the function to "Install VMware Tools".  For Fusion, refer to the manual instructions at, for Workstation refer to, and for ESX refer to

Which ever method you use, you will need to execute the following commands to install VMware Tools in your CentOS machine:

mount /dev/cdrom /mnt
cd /tmp
tar zxvf /mnt/VMwareTools-*.tar.gz
cd /vmware-tools-distrib
umount /mnt
rm -rf vmware-tools-distrib

Make Networking work after Cloning

If you are building a VM for cloning, you will want to execute the following commands to make networking work when the VM is started after a clone:

sed -i "/UUID=.*/d" /etc/sysconfig/network-scripts/ifcfg-eth0
sed -i "/HWADDR=.*/d" /etc/sysconfig/network-scripts/ifcfg-eth0
rm /etc/udev/rules.d/70-persistent-net.rules

If you start your base image again for any reason after this step, you will need to execute the last line of the commands above as the referenced file is generated after every boot.


At this point you can clone the VM you built above as many times as you need.  After starting up the cloned VM, you will want to edit the host name using the following commands:
If you need to change the host name for the machine you are working on, you can do so with the following commands (either replace $NEWFQDN with the new fully qualified host name, or set this variable before executing the commands):

sed -i 's/HOSTNAME=.*/HOSTNAME=$NEWFQDN/' /etc/sysconfig/network
hostname $NEWFQDN

If you are using avahi, you need to restart it to get the new host name picked up:

service avahi-daemon restart


Popular posts from this blog

Ghetto Cloud Foundry Home Lab

Using Snapshot Isolation with SQL Server and Hibernate

Fedora, Ant, and Optional Tasks