Adventures with chef-server and Gentoo, part 3

Continuing from Part 2

The next step comes in a bundle of two. I am targeting the Rackspace Cloud platform. Each Rackspace slice gives you two ethernet devices. eth0 points to a public IP address and eth1 gives you a private IP address in the 172.* range. Packets going through eth0 incurs a charge while packets routed through eth1 does not. Since the default, non-nanite install of Chef assumes you will be polling chef-server every so often, having lots of these slices running around gets expensive if routing through the outside addresses.

So what we really want is a way to reference each of the nodes by the private IP address. There are several ways to do this. Since these are non-public addresses, you cannot use the Rackspace DNS server nor any of the public DNS services. 37Signals uses djbdns; others have suggested PowerDNS. The lowest-barrier-of-entry solution I have found so far queries the server index and builds a /etc/hosts. I chose this solution, with some modifications.

First, the /etc/hosts file emitted by the recipe described in that blog post maps the FQDN to 127.0.0.1. That's not what I want. If I want to use 127.0.0.1, I want to reference it with localhost and not the FQDN. Years ago, I remember reading how the Linux loopback device performs insanely faster than going through a regular ethernet device stack. I don't know if that is true now or where I can find the performance figures for it. Regardless, what I don't want is the ip reference for a FQDN changing on me. I have started a Rackspace cookbook here and dropped the ghetto DNS recipe in there along with my modifications.

Second, I do not fully agree with the Opscode design decision to use Fully-Qualified Domain Names. Philosophically, I'm adopting the Chef infrastructure as part of a deliberate strategy based on the principles of extropy. In short, stable forms arises from unstable substrates, and when the economics of the unstable substrate reaches a critical point, the substrate becomes disposable. They don't matter anymore to those who are operating at the higher abstraction. The fallout of that is that the unstable substrate supporting the stable abstraction becomes anonymous.

In practice, I want anonymous server VMs that can turn into anything, very much like pluripotent stem cells. I'd rather have the authoritative Chef node id be some sort of random md5 hash. If I wanted named servers, I'd do that on top of the anonymous nodes by using tags. I don't really want to manage which node is db0 and db1; by keeping these anonymous, I can create parameterized infrastructure. What we have planned at Sparkfly include an architectural building block called a "segment", which has a well-defined set of components spread across a group of two or three servers. Rather than specifying each node and each role for node, this should be all done programmatically using a higher-level description.

On the other hand, we will always have certain servers that must always be referenced. One would be chef-server. I would need a site-wide hosts table referencing that. (But even this can eventually be built on top of anonymous nodes).

Looking through the docs, there were some extra hoops I'd have to jump through to set the node name differently from that of the FQDN. For now, since I'm simply trying to get a working infrastructure up and running before I start digging in and trying to make more sweeping changes, I wrote a kludge that creates an anonymous name and appends that to the /etc/hosts. There is another supporting Gentoo script added into /etc/conf.d/local.start which detects the presence of a file, /etc/conf.d/bootstrap and runs kludge script. Run it, and you get something like node-1ee5ddee46d52c269b898bf216

What's next are rake tools to help query and reference based on tags. I also reserve the right to change my mind. Maybe I'll get tired of copying & pasting non-sensical md5 hashes when working with knife. After all, strategy is pragmatic and concerns what is, rather than its idealized form.