A Beowulf Cluster
So far, we have played around a little bit with the idea of a stand-alone XBOX doing some neat things. But what if we want to actually make a high availability processing cluster across all of our Linux machines at once? This is the murkier world of XBOX PCs, distributed computing. There are a lot of really good documents detailing how to set up a secure, robust and stable Beowulf Cluster, but this isn't one of them. We only want to benchmark 8 XBOXes in parallel operation.We started by finding a good location for our XBOX cluster to run. A few 2x4 wood boards stacked together on the lab floor were good enough for us, apparently. Just to get an idea of how big our cluster was going to be, we stacked the XBOXes up in two columns of four as you can see below. It is a good idea to have plenty of room on all 4 sides of your cluster so that you can easily check cabling and such.
Click to enlarge.
A 16 port gigabit switch provided us with network connectivity. Unfortunately, we are not actually using gigabit connectivity since the XBOX network devices only operate at 100Mbps. A 100Mbps switch usually runs for about $50, although it isn't wise to go too cheap on the largest bottleneck in the cluster. We crimped our own cabling as you can see in the image below. Even though we are just hacking a system together, keeping the cabling organized is very important since diagnosing a bad system will be nearly impossible if wires are all over the place.
The next step was actually placing the chip in each machine and removing the hard drive so that we could image it. Modifying each of the XBOXes took some time. It took us a few hours to get all of the mod chips in correctly and start up the SmartXX BIOS. By default, SmartXX does not boot to Linux, so we had to tell the BIOS after each chip was inserted.
Click to enlarge.
First, we installed Linux on the master XBOX machine. This machine will eventually become the master of the cluster, but we will use it as a template for the other machines before we clone the hard drive onto the other XBOXes. All of the programs that we planned on using, such as distcc, djohn, etc., were installed on this template machine.
We will need to identify each of our XBOXes in the hosts file. This will allow us to identify each machine without doing a DNS lookup on its alias. Our hosts file looks like this:
| master# cat hosts | |
| ::1 | localhost |
| 127.0.0.1 | localhost |
| 192.168.1.10 | master |
| 192.168.1.11 | slave1 |
| 192.168.1.12 | slave2 |
| 192.168.1.13 | slave3 |
| ... | |
By using a hard drive blaster, we able to image all of our other XBOX hard drives in about an hour. Using Ghost or another drive imaging utility will work as well, but you can only do one or two drives at a time instead. If you are really desperate (or cheap), installing Linux on each machine manually works, but leaves the most room for error. Right now, we have eight XBOXes that are exactly identical in every respect. We need to designate one of these to be the master, so the first machine is booted up for a little bit of configuration. Dhcpd, the program that will assign IP addresses to each of our slave machines, was configured as such:
# dhcpd.conf # option domain-name "cluster"; option domain-name-servers 192.168.1.200; option subnet-mask 255.255.255.0; default-lease-time 3600; max-lease-time 86400; ddns-update-style none; subnet 192.168.1.0 netmask 255.255.255.0 { range 192.168.1.11 192.168.1.17; option routers 192.168.1.1; } We really don't care which IP gets assigned to which machine; although, if we boot each XBOX in sequential order on the rack, their IP address will line up with their position in the boot sequence.
ncG1vNJzZmivp6x7orrAp5utnZOde6S7zGiqoaenZH52f5hobw%3D%3D