Ghetto Cloud Foundry Home Lab

Over the past few months, I've been cobbling together my own Lab to be able to gain experience with Cloud Foundry.  Sure, I could have gone the much simpler route of bosh-lite, but I wanted to get a broader set of experience with the underlying IaaS layer in conjunction with working with Cloud Foundry.

My lab hardware was purchased from various places (eBay, Fry's, etc) when I could get deals on it.

Rocking the Ghetto Lab
At a high level, the hardware looks like this:

Machine CPU Memory Storage Notes
HP Proliant ML350 G5 2x Intel Xeon CPU E5420 @ 2.50GHz 32 GB (Came with some disks, but mostly unused) vSphere Host 1, Added a 4 port Intel 82571EB Network Adapter
HP Proliant ML350 G5 2x Intel Xeon CPU E5420 @ 2.50GHz 32 GB (Came with some disks, but mostly unused) vSphere Host 2, Added a 4 port Intel 82571EB Network Adapter
Whitebox FreeNAS server Intel Celeron CPU G1610 @ 2.60GHz 16 GB 3x 240GB MLC SSDs, in a ZFS stripe set, plus spinning disks for home files storage Already in place for home storage. I added SSDs to host VMs, and a 4 port Intel 82571EB Network Adapter
Netgear ProSafe 16 Port Gigabit Switch Storage Switch for running Multipath iSCSI between FreeNAS and vSphere Hosts

I'm running vSphere ESX 5, and I'm using the vCenter Appliance to manage the cluster.

The vSphere Hosts and FreeNAS server are all on the same network as my personal devices since these machines provide some services beyond a lab for my Cloud Foundry work.

Installing vSphere to these boxes was quite simple because HP and the Intel Network adapters are on the compatibility list for vSphere.  I highly recommend you check out http://www.vmware.com/resources/compatibility/search.php if you are trying to build your own lab with other components.  There are ways to get other components to work.  They usually involve creating custom install packages for vSphere to inject the appropriate drivers.  I didn't want to go that route, so I made sure to pick stuff that was on the compatibility list.

I then deployed the vCenter Appliance VM to one of the hosts, and made sure to set it to start up automatically with the host, just in case my UPS caused my hosts to shutdown due to a power outage.

vCenter is running as a VM inside a vSphere host...and managing that vSphere host.  Inception. ;)
Inside vCenter, I've defined a Distributed Switch with a port group that is uplinked to my home network, 4 port groups for multipath iSCSI to the FreeNAS server for storage, and then 1 port group with no uplinks as a private, virtual network segment.  I created private port group mainly because my home network is just a 24 bit net mask (255.255.255.0) with about 254 usable addresses, and I didn't want to have my Cloud Foundry VMs fighting with my home devices for IP addresses.  The network on the PG-Internal port group is using a 16 bit net mask (255.255.0.0) for around 64K addresses per network.

Distributed Switch and Port Groups
To allow VMs on the PG-Internal port group to access the outside world and to be accessible in a controlled way from my home network, I created a minimal Ubuntu VM that has a virtual network adapter connected to the PG-FT-vMotion-VM port group that uplinks to my home network, and another adapter connected into the PG-Internal port group to route packets between my home network and that private network.  I then configured Ubuntu to forward packets and act as a NAT for the network on the PG-Internal port group by loosely following the instructions at http://www.yourownlinux.com/2013/07/how-to-configure-ubuntu-as-router.html.  The differences there are that I didn't need full setup that was used for iptables on that page, and I added "dnsmasq" to that box to cause the IPs for Cloud Foundry on that network to resolve to my internal IPs.   More on that in a later post.

At this point, I chose an IP in the network on the PG-Internal port group that I was going to use for the HAProxy that Cloud Foundry was going to use.  I noted this IP, and used it in the subsequent network setup steps, and for the install of Cloud Foundry.

Finally, my home internet router is the default gateway for my network, so I made sure to add a static route to my router to route packets to the address of my Router/NAT VM so that apps in Cloud Foundry could get out to the internet if needed.  I also set up port forwarding from my internet router to forward port 80 and 443 to address of my Router/NAT VM on the 24 bit subnet so that I could access my Cloud Foundry install from the outside world.  Finally, I needed to setup port forwarding on the Router/NAT using iptables to forward requests coming from the outside via my internet router to the HA Proxy IP address (in the 16 bit subnet).  I was able to do that with the following two iptables rules (you would need to set the right IPs for your own networks, of course):

-A PREROUTING -d 192.168.X.X/32 -i eth0 -p tcp -m tcp --dport 443 -j DNAT --to-destination 172.X.X.X:443
-A PREROUTING -d 192.168.X.X/32 -i eth0 -p tcp -m tcp --dport 80 -j DNAT --to-destination 172.X.X.X:80

Oversimplified view of the network config
After getting all of this set up and working with VMs, I use the Pivotal Cloud Foundry distribution to deploy Cloud Foundry to my vSphere Lab.  I just went to http://network.pivotal.io, signed up for a free account, and then was able to download the latest Ops Manager OVA, and Elastic Runtime package from https://network.pivotal.io/products/pivotal-cf.  I then just followed the instructions starting at http://docs.pivotal.io/pivotalcf/customizing/deploying-vm.html to deploy the OVA to vSphere.  I made sure to attach that VM's network adapter to the PG-Internal port group so that it would be able to install Cloud Foundry to that network.

You will need a wildcard DNS entry defined to be able to access the elastic runtime component of Cloud Foundry.  The docs give you some tips on how to use the xip.io service do this, which is probably the easiest.  I hate doing things the easy way, I had my own public domain already, and I knew I wanted to be able to access my Cloud Foundry install from outside my home so I setup a wildcard DNS entry to use when I was out of my home.  I used No-IP to make sure I could get my dynamic, ISP provided IP address registered, and then I used my domain name registrar's DNS web interface to add a wildcard DNS CNAME record that pointed to that No-IP dynamic address, which pointed to my router.  In this way, if I browsed to my own domain name entry, I would get sent to my home router's public IP, which would then be NAT'ed to my internal network, which would be NAT'd again by the Router/NAT VM to my private Cloud Foundry network.

By using DNSMasq on my Router/NAT VM, I was able to put the same wildcard DNS entry into DNSMasq, and have it be the DNS server for the private network.  This is important because you need to make sure that the BOSH Director VM and any errand VMs that it creates can access the HAProxy instance VM that BOSH provisions.  Without this, I was having trouble getting packets to route properly from VMs in the private network to other VMs in the private network using the public DNS entry.

After configuring all the settings in the Ops Manager web UI, I was then able to add the Elastic Runtime download and configure that tile as well.  Then, it was simply a matter of waiting for the install to complete, and then signing in to the console.

Boom!  Working Cloud Foundry.
There is a great deal more detail I want to share, but it was too much for a single post.  I'll break out the detailed information into separate posts, based on interest.  Please comment and share your experiences and requests for more detail.

Comments

Sani T. Chabi Yo said…
First of all thanks for the post. I'm planning to build my own Home Lab as well. I have a quick question, how much did you spend on the whole setup?
February 24, 2015 7:48 AM
Chris DeLashmutt said…
It is hard to say what I totally spent. I already had some parts in place like the FreeNAS server, and router. The parts I needed to get for this build were the storage to hosts interconnect switch, the HP servers, additional memory for the servers, and 2 additional 4 port Intel NICs. All total, I would say that ran me $800-$900.

You can do this more cheaply by only using one host, local SSDs in that host for storage, and you would be able to eliminate the NAS, storage switch, and even the Intel 4 Port NICs. That is not a bad way to get started with something more cost effective.

The main trick with a limited set up like this is that you would just set up a cluster in vCenter with that single host in it. The only downside to that single host set up is that you wouldn't be able to play with vSphere HA/DRS if you wanted to. Since I was planning on using my lab for other purposes as well, I wanted to spend the extra money. :)
February 24, 2015 4:40 PM
Edward Wong said…
Hi, how do you managed to use 82571 on FreeNAS?
I have the Intel Pro1000 PT Dual Port (same chip), and tried FreeNAS/NAS4Free, a high traffic will immediately cause a machine reboot (since I have onboard Broadcom card and works very well if I do not pass traffic thru Intel card)
March 03, 2015 4:19 AM
Post a Comment

Popular posts from this blog

Using Snapshot Isolation with SQL Server and Hibernate

In the course of working on some deadlock issues, I found that MS SQL Server exhibits some unexpected (to me, at least) locking behaviors that can affect performance and cause deadlocks. In the end, I found that SQL Server provides an isolation level called Snapshot isolation which removes the need for locks on rows and removes lock contention. To use this isolation level, you need to execute the following SQL on the database that you want to use the isolation level with (replacing MyDatabase with your database name, of course): ALTER DATABASE MyDatabase SET ALLOW_SNAPSHOT_ISOLATION ON ALTER DATABASE MyDatabase SET READ_COMMITTED_SNAPSHOT ON Next, you need to modify your hibernate.cfg.xml file to add a property to tell Hibernate to use the Snapshot Isolation level: <!-- The 4096 isolation level is the setting to use with the jTDS or Microsoft JDBC drivers --> < property name =”hibernate.connection.isolation”>4096</property> Snapshot isolation is not a panacea...
Read more

FreeNAS, CIFS, and user login problems

Image
I have been in the process of converting from the dead-end Windows Home Server product to a FreeNAS server for my home NAS.  I was trying to replicate the same sort of set up that WHS had for user directories.  In WHS, there was a share called "Users" and under that share there were various user directories corresponding to each user account.  I thought it would be pretty simple to replicate that same behavior to allow users to simply attach to \\nas-server-name\Users in Windows, and then navigate to their own user folder.  What follows is how I achieved this, as well as a problem I fought with for a while that prevented certain users from connecting via CIFS. First, I started off the process by installing FreeNAS to a USB stick, added a few disks in the server, plugged in the USB stick and booted up.  I performed basic config for the networking, and admin account passwords, and turned on the CIFS service. Next, I created a some user accounts fo...
Read more

Official Pandora Desktop and Hotkey Support?

EDIT: There is a slight problem occasionally with this script not being able to send keys to the Pandora Desktop Air app. I've got an updated version of the script that you can use instead of this one. Pandora is a different sort of online music service that uses the work of the Music Genome Project to automatically selects music similar to an artist or particular song, and then streams it over the Internet for your listening pleasure. You can't select individual songs, but you can skip ones you don't like and make as many music channels as you like to suit your particular tastes. Pandora provides a free version of the product that is supported by visual advertising, and a very inexpensive ($36 USD per year as of the creation of this entry) subscription supported version of the product that provides higher quality streams and no advertising. Traditionally, Pandora has been a Flash-based application that required you to have your bulky browser open to listen on the deskto...
Read more