Remarkable 4.0.0.alpha2

Remarkable 4.0.0.alpha2 now released. This release reorganizes and simplifies loading of the component. Here's how you install it:

sudo gem install remarkable_activerecord --pre

Add Remarkable to your Gemfile:

group :test do
   gem 'rspec', '>= 2.0.0.alpha7'
   gem 'rspec-rails', '>= 2.0.0.alpha7'
   gem 'remarkable', '>=4.0.0.alpha2'
   gem 'remarkable_activemodel', '>=4.0.0.alpha2'
   gem 'remarkable_activerecord', '>=4.0.0.alpha2'
end

In spec/spec_helper.rb add:

require 'rspec/rails'
require 'remarkable/active_record'

If you just want Remarkable core, use:

require 'remarkable/core'

If you just want Remarkable::ActiveModel, use:

require 'remarkable/active_model'

All locales inside the gem are now automatically loaded. So if anyone wants to provide a translation for say, Traditional Chinese, feel free to fork and send a pull request.

This release also fixes the bug in alpha1 where the English localization was not properly loading for remarkable/active_model.

Changes for Macro Developers

Although I wrote about improving macro registration, I googled around some more and found "Gem Packaging: Best Practices". The system I was proposing would overlap the functionality of Rubygems and Bundler (in so far as dependency checking). It would have been over-architected.

Requiring remarkable/active_record should automatically require remarkable/active_model, which should in turn require remarkable. The version dependencies themselves are described in the gemspec file. Further, users should just be able to require a single file without having to mess with Remarkable.include_matchers! to get things done.

By moving the files from remarkable/lib/remarkable to remarkable/lib/remarkable/core, it opens it up developers to add to the remarkable namespace. For example, we can have a require 'remarkable/foreigner' by making a lib/remarkable/foreigner.rb inside the gem and dropping the rest of the files into lib/remarkable/foreigner/.

Instead of using lines such as

# Load Remarkable ActiveModel files
dir = File.dirname(__FILE__)
require File.join(dir, 'remarkable_activemodel', 'base')

[ src ]

we just have

# Load Remarkable
require 'remarkable/core'
require 'remarkable/active_model/base'

[ src ]

But how do we avoid loading the system gems when we want to run tests? We throw custom load paths into $LOAD_PATH or its alias, $:

# In spec/spec_helpers.rb
$:.unshift(File.dirname(__FILE__), '..', 'lib')

To help load these paths, I've added a path_helpers.rb that you can take a look at.

For details, see remarkable_activemodel and remarkable_activerecord for examples on how the loading is done.

The knife is out for ruby ... ORLY?

Warning: Link Bait

From Hacker News, comes "The Knife is out for Ruby". "Dynamic languages are coming to the end of their honeymoon period. This is bad for Python and maybe PHP, but it is nothing short of a disaster for Ruby."

... leaving aside the religious war of Static vs. Typed language, how about viewing this from the point of view of Pragmatic Extropy?

"Type safe static languages like C#, Java and Managed COBOL are much more challenging to work with, or at least they were."

... X is better because it is more challenging to work with. Does that mean that the programmer has to be smarter?

"However, the gap is closing or may have completely closed. Type inference, generics and much better IDEs have made static languages really easy to work with. The permissive nature of dynamic languages is only really treating the symptom that programmers do not want to have to remember and key-in everything's type."

... X is better because it forces you to think through about issues Y?

Scrolling down to the comments, the author reveals a bit about himself:

"My day job is in a compiler writing team. I spend a lot of time in a lot of different languages. My view is that dynamic languages make sense for code bases which do no justify heavy tooling and which are not going to be around for a very long time. I used to think that dynamic languages were a lot more useful than that because of productivity - but recent advances in IDEs and type systems have convinced me that even if it is not the case now, over the next few years the difference in productivity will diminish a lot."

I was talking to folks on #pragmatic_extropy about my experience geocaching with a friend. I watched my friend get so dependent on the GPS, missing out obvious environmental cues and clues. I promised myself that I wouldn't do that if I held the GPS in my hand. My friend got a call and handed me to the GPS, and within five minutes, he was pointing out cues I missed because I was navigating with the GPS.

The GPS had easily become an extension. The neural net of the brain adapts to the external interface, in which the external interface becomes part of the complete circuit. There's a paper written about the phenomena of two neural nets forming an internal language idiosyncratic to those two particular neural nets. You can see it in two intimately-close people who have been around each other for a long time. Likewise, I remember an article about how family groups would socially (and unconsciously) assign specialty knowledge to specific family members. The same is true for any tool we use -- tab completion on the command line for example. I don't feel right without being able to hit the tab key since I remember filenames more on the first few letters and the number of tabs I have to hit. Other tools include code snippets, VIM cursor controls, distributed version control ... and IDE-based type inference and whatnot.

Some languages, however, such as Lisp, has the rare ability to transform how we think because the language is powerfully expressive enough. Its power derives from having very little syntax and does not require external tools to grok. Someone programming in .NET, for example, require an IDE to be productive because the IDE forms the complete neural circuit. Something like Lisp (within certain cases) has the complete neural circuit held inside the head. There is more transformative value for a Lisp-user than a .NET-user, even with a better IDE. While it can be argued that the IDE closes the gap in terms of productivity, it does not close the gap in terms of arete, though it may go with the grain of extropy.

Adventures with chef-server and Gentoo, part 9

Continued from Part 8

So my nifty Ghetto DNS for Rackspace Cloud backfired on me.

I spent several hours (!?) trying to figure out why (1) the recipe works on my dev environment, (2) the recipe works at our in-house Xen Cloud Server environment, and (3) the recipe refuses to work on the Rackspace Cloud Server.

I had narrowed it down to the fact that:

search(:node, "rackspace_private_ip:#{rackspace_hosts[:private_net]}" ) do |n|
  ip = n[:rackspace][:private_ip]
  hostnames = [ n[:fqdn] ]
  hostnames << (n[:rackspace][:private_aliases] || []).sort
  hosts[ip] = hostnames.flatten
end

... was not in fact pulling down node[:rackspace][:private_aliases]. It was not setting it from the override_attributes of the roles properly. It was ignoring what I had set with knife node edit. It should have tipped me off that every time I ran chef-client, node[:rackspace][:private_aliases] was getting overwritten. Instead, I focused on the fact that I had compiled ruby with threads enabled, and maybe, just maybe, chef-solr-indexer had corrupted solr or something. (It didn't). I ended up wiping the solr data directory and forced a reindex ... and it still came out the same.

Finally, out of sheer flailing around, I finally saw a detail I had missed in the growing red haze of frustration. The Rackspace server had a "public_ip" attribute set that I had not set at all. Where did that come from? Suspicion blossomed. I ran ohai on the Rackspace Cloud server, and here is the lesson learned:

ohai manages the rackspace namespace if you are on Rackspace Cloud. Don't touch it!

I have no idea whether this had always been there or not, though I did notice a knife rackspace option pop up since 0.8.16. The fact that ohai will automagically detect Rackspace private ip is awesome -- I don't have to use my own version, except when I'm trying to emulate Rackspace. I'll probably split that out into its own recipe, and use the :ghetto_dns namespace for the host aliases instead.

Another action item: I will be more assertive about asking Opscode for a Changelog -- assuming they are not keeping up with it. And assuming that I ever get out of this Chef iteration I'm not supposed to be operating in...

But for those of you itching to develop your own recipes for Rackspace, now you know.

Adventures with chef-server and Gentoo, part 8

Continued from Part 7

Although in Part 7, I said that was the end of the Chef iteration for now, I wanted to finish up just one more thing. (It's always just one more thing). I attempted to install a fresh chef-server for our internal lab environment, now that I've finished up with the recipes on my dev environment. I followed through the notes I took in Part 2, and proceeded to run into a Big Surprise.

knife configure --initial asked for the location of webui.pem, which I promptly dismissed and ignored because, well, what does the webui have anything to do with setting up an admin use for knife. Well, apparently it does. I went to #chef when my attempt failed to create an access key, at which point, kallistec and jtimberman pointed out what has changed in 0.8.16:

1. chef-validator and its access key, validation.pem are no longer admin users. The only thing they can do is to create a new non-admin access keys. That's good because that is one less thing to worry about (having to be disciplined enough to delete /etc/chef/validation.pem when booting up a new node).


2. It does mean that you have to have access to an admin access key in order to create yourself an admin key for knife


3. Unfortunately, using webui.pem makes it sound as if the webui is repurposed to creating new admins. I asked the guys, what if I don't want to install webui?. Their reply is that webui.pem is generated by chef-server-api, rather than chef-server-webui. That makes sense.
   
   What doesn't make sense is naming it webui. I thought something like api-master-key would be more descriptive.


4. The flip side of it though, is that kallistec thought there would be even more trouble had that not been used. That's probably true, because my dev upgrade from 0.8.10 to 0.8.16 was transparent.


5. Shouldn't knife configure -i drop a non-admin user anyways? I thought about that and realized that I would not have liked it to silently create a non-admin user and have all of attempts at uploading the cookbook fail. At this this way, it is obvious where the problem lie.


6. I wasn't too happy, but at the end of the day: this is open source software and people are putting their hard work out there, and this is still bleeding edge code with lots of things subject to change. Opscode may change it in the 0.9, which makes sense considering we'd all expect lots of breaking changes in that release.


7. Also, the only reason I even came across this was because I was (wait for it...) using a non-standard installation of knife. I wanted knife configured on my dev box used to control deployment against the lab site, so I didn't see a point in configuring knife on the server. The knife configure -i, when run on the chef-server and taken with default, would have just worked.

So I ended up with this kludge:

$ knife configure -i
Overwrite /home/hhh/.chef/knife.rb? (Y/N) Y
Your chef server URL? [http://localhost:4000] http://chef-server.lab:4000
Select a user name for your new client: [hhh] hosh
Your existing admin client user name? [chef-webui] 
The location of your existing admin key? [/etc/chef/webui.pem] /home/hhh/.chef/api-master-key.pem
Your validation client user name? [chef-validator] 
The location of your validation key? [/etc/chef/validation.pem] /home/hhh/.chef/lab.validation.pem
Path to a chef repository (or leave blank)? 
WARN: Creating initial API user...
INFO: Created (or updated) client[hosh]
WARN: Configuration file written to /home/hhh/.chef/knife.rb

I had more unrelated trouble, such as not configuring /etc/hosts right and ended up having to enter this stuff three or four time. Maybe when I have time, I'll hack up something that lets you take an existing configuration and create a new admin key without having to type all of the above again. Scratch that itch. (And get off my lawn!)

Adventures with chef-server and Gentoo, part 7

Continued from Part 6

Earlier this week, I integrated veszig's's Portage code into my set of cookbooks. I've also merged in veszig's keywords, use, etc. provider into a seperate module. Along the way, I cleaned up some things (my fascination with Scheme is showing through in the code) and made it so that it will default to EIX if available and fallback to emerge --search if not. I monkey-patched it into the Portage Provider, and may submit it as a real patch upstream to Opscode.

Some gotchas I ran into in the past few days:

1. Chef 0.8.10 - knife does not properly dump metadata.json, resulting in a rebuld everytime I upload a cookbook.


2. Chef 0.8.14 - This release fixes the bug in 0.8.10, and introduces another one: it does not properly set the file mode. (Or more precisely, it looks like it creates a file in a read-only mode and did not set it as specified, or something).


3. Chef 0.8.16 - This release fixed the mode bug. However, it doesn't work with json 1.4.3.

Why do I know this? I was working on the Improved Ghetto DNS recipe. It figure out what the private IP is and save that back into the sever index. jtimberman had said you did this with node.save (huh...), then suggested dropping it into a ruby_block so it is saved at execution time rather than compile time. When I thought about it though, I decided to do this at compile time instead and put guards to keep it from saving the information. This stuff touches host names, the fundamental stuff that Chef assumes you've already figured out. There is (unfortunately) a lag between the time a private ip gets reported, and when it gets indexed by the server index, so we essentially have an ad hoc eventual convergence, over multiple runs of chef-client. Yes. That's why it is Ghetto DNS.

When Ghetto DNS drops the new /etc/hosts file, chef-client continues on its merry way. Unless it is set to mode 0600, of course, in which case, it doesn't know where chef-server is. Oops. Chef 0.8.16 fixed that. Too bad there is no chef-overlay that pulls that in, so I ended up updating with rubygems.

I am probably going to create Chef gem upgrade recipes at some point. However, I've come to the end of the iteration, so I'll need to work on a different project. Maybe I'll update more Gentoo stuff in my off hours, at least until the next Chef iteration comes up.

As it is, I've gotten a lot of monit recipes up. One gotcha, at least for 0.8.10, was that nested definitions do not properly pass params, so you had to rebind variables locally within the block. As it is, I created several monit macros:

monit

Basic monit macro.

Example:

monit 'my_nginx_site'  do
  cookbook 'my_site'
  source 'nginx.monit.erb'
  variables(:listen_ips => listen)
end

monit_service

Monit a background service

monit_service 'chef-solr-indexer' do
  process :pid_file => '/var/run/chef/solr-indexer.pid',
    :timout_before_restart => '30'
end

monit_net_service

Monit a service listening on an IP port

monit_http_service 'couchdb' do
  process :listen_ips => [%w(127.0.0.1 5984)],
    :pid_file => '/var/run/couchdb/couchdb.pid',
    :timout_before_restart => '30'
end

monit_http_service

Monits a HTTP service, checks HTTP protocol

monit_http_service 'chef-server-webui' do
  process :listen_ips => [[nil, '4040']],
    :pid_file => '/var/run/chef/server-webui.4040.pid',
    :timout_before_restart => '30'
end

monit_service, monit_net_service, monit_http_srevice all take the following arguments:

process

• :start_cmd - override the start command


• :stop_cmd - override the stop command


• :pid_file - override the pid file


• :timeout_before_restart - override the timeout

cookbook

Specify the cookbook to pull the source template from

source

Specify the source template to use

variables

Specify the variables to pass to the template

enable

If true, then create the monit configuration file.

For complete source code, see it here.

With these, I created chef::monit_server, chef::monit_server_webui, chef::monit_solr, chef::monit_solr_indexer, couchdb::monit_server, and added monit to gentoo::portage_rsync_server and gentoo::portage_binhost_server. Weirdly enough, the one that gave me the most trouble is rsyncd, since it does not behave very well as a daemon. I still have not figured out how to monit rabbitmq, on account that the stock rabbitmq-server does not drop a pid file. (However, rabbitmq-multi does).

It was loads of fun randomly killing off processes and watching them come back up.

Finally, Seth at MaxMedia told me about CloudKick at the Atlanta Ruby User's Group meetup. It is too much for me, but I suppose if I were doing something that I needed to show investors or enterprise customers that "We Can Scale", that would be it.

Idea: Cost Accounting using Chef

Since Chef describes your infrastructure as a set of resources, I think it is possible to go beyond the basic bandwidth + CPU/hours for cost accounting.

Suppose you have a set of heterogenous web sites, each using a different classification of recipes:

• Site A: static files


• Site B: static files


• Site C: PHP-FCGI (Wordpress)


• Site D: Python (Trac)


• Site E: Ruby on Rails 2


• Site F: Ruby on Rails 3

In our hypothetic infrastruture, Sites A and B are served from a single Nginx server. Site C and D are also served on that same Nginx server, using proxies.

Site E & F requires a stand-alone web app, however, they both share the same Master MySQL database (though the database schema are seperate).

Now, how do you do the cost-accounting for that? If we were to use basic bandwidth + CPU/hours, the best we can do is determine what each of the hosts, in this case, 4 hosts, costs us. This assumes you are using EC2 or Rackspace CS.

On the other hand, we know from our Chef-repo describing this whole setup, that Sites A, B, C, and D each are Host 1 and split the costs of Host 1. We can weigh Sites C and D higher since they are not using static files, and therefore doesn't take advantage of Linux send_file.

Sites E and Site F each have Host 2, and 3 respectively. However, they would include the costs for sharing Host 4, used as the MySQL master.

I'm not exactly sure how this would be implemented. We can query this by node, but better yet, if we have a higher-level description of each of those sites, we can calculate cost structure around that.

Ultimately, this doesn't matter as much if your site runs a single app exclusively. But even a large company, such as Flickr or Facebook with many different subsystems, would want to be able to track logical cost structure of each subsystem, versus how much revenue each subsystem brings.

Idea: Rubygems for Chef Cookbooks

One day (but not today), we will need the equivalent of Rubygems for Chef Cookbooks. Or at the very least, site cookbook collections.

Wireless Virtual Machines

Problem:

• I don't like doing dev work on OSX, so I have a number of VMWare vms running Gentoo Linux where I do my actual development. OSX works great to provide anywhere-access via wireless and sleep.


• Most free wifi places such as Panera's, Border's, and Barnes & Nobel requires you to access their website first before they let you through.


• Besides, I don't like using VMware to bridge connections for Linux through WiFi.


• I don't like using a NAT since I'm often already behind a NAT


• I still need wireless net access when I'm sitting at a coffee shop

Solution:

1. Install and configure nylon via MacPorts


2. Install and configure proxychains on Linux VMs.

Done.

Alternatives:

• I have tried antinat but it was way too complicated and I could get it to play without any authentication.


• When I was using a Gentoo linux laptop for my primary dev platform, I have tried using tsocks and it simply didn't work very well. proxychains, on the other hand, works.


• You definitely want to lock down nylon so that it only accepts inbound connections from the vmnet1 device


• I have an outgoing ssh proxy for WiFi anyways. Guess what proxychains does?

Adventures with chef-server and Gentoo, part 6

Continued from Part 5

Lots of updates:

1. I further refined the portage_conf definition. You can pass :force_regen to force a regeneration of make.conf. This is necessary in the case of mirrorselect if you need to install a package.



2. I added a bunch of recipes. Here is an index of them so far:
   
   

   
   

   portage

   
   

   Base portage

   
   

   chef_overlay

   
   

   Installs veszig's chef-overlay

   
   

   feature_buildpkg

   
   

   Enable building binary packages

   
   

   feature_getbinpkg

   
   

   Fetch binary packages from binhost

   
   

   feature_nodocs

   
   

   Do not build docs, man pages, and info pages

   
   

   cron_emerge_sync

   
   

   Adds emerge --sync to cron for nightly-syncs

   
   

   exclude_categories

   
   

   Exclude certain categories from sync, such as nethack ...

   
   

   portage_binhost

   
   

   Point to a site binhost

   
   

   portage_rsync

   
   

   Point to a site rsync mirror

   
   

   mirrorselect

   
   

   Selects the three fastest Gentoo Mirror.

   
   

   gentoolkit

   
   

   Installs gentoolkit

   
   

   portage_binhost_server

   
   

   Sets up a binhost server with Nginx

   
   

   portage_rsync_server

   
   

   Sets up a local rsync mirror

   
   




3. I talked to jtimberman and veszig on #chef@freenode.net and got some feedback:
   
   
   
   • jtimberman suggested Resource/Provider to replace portage_conf. That was the direction I've been heading towards, though what I have now works. (Unless you want to notify/subscribe to it, then it doesn't)
   
   
   • veszig showed me what he has in gentoo-cookbooks. He is using eix for package queries, and actually has a LWRP for emerges. I think it is called gentoo-package. It will let you set USE flags and masks, unmasks, and keywords. I've been using definitions, but again, I can't send notifications to them. I think the way he has set up the emerge is better than what I was going to do -- monkeypatch the Portage provider
   
   
   • jtimberman mentioned that knife can do Rackspace deploys, though it is Debian/Ubuntu centric. But the are accepting patches :-) He also mentioned more things about the data-driven database and application recipes, which I can't wait to get started working on that.
   
   
   
   

All in all, good times. I'm going to attempt a merge of my stuff into a fork of gentoo-cookbooks, and move anything generic back into the mainline cookbooks repo. I may still end up submitting an improved Portage provider, just in case you want to still remain distro-agnostic.

Addendum

It has been two months since I wrote "Intangible Assets and Sunk Cost Fallacy", and this conversation with jtimberman and veszig illustrates the most noticeable payoff of intangible assets, at least in the Rails world. The fact that I can look at veszig's code and veszig can look at my code meant that we can exchange best practices faster. I didn't know about eix, and while more complicated, portage_conf is pretty useful. This would be much more difficult to do if we were exchanging words. Neither of us can guarantee that we are following the exact steps as specified (something that, I'm sure, eats up the vast majority of systems troubleshooting). On the other hand, we each can find out what the other did, simply by looking at the code. "Best Practice" becomes self-documenting, executable code, code that can be conserved, exchanged, and source-controlled.

Ruby, Python, Perl, and Java each have a vast, vibrant community because of the libraries of gems, eggs, CPAN, and Java code; I'm glad to be a part of something new here -- a library of infrastructure code that is still largely invisible to the mainstream. It's my "secret" competitive advantage.

Data Driven Application Deployment with Chef - Blog - Opscode

From Opscode: Data Driven Application Deployment with Chef - Blog - Opscode.

I was wondering when they were getting around to this. As I mentioned in a previous post, we're going to use "segments" at work, and that would be data-driven. In our case, it would be Rackspace Cloud Servers, not EC2 and Rackspace Cloud Files, not S3, but the principles remain the same. The databags seems like a great place to drop the descriptions in there, though I am not sure if I want to have it requery the index every 30 minutes.

Adventures with chef-server and Gentoo, part 5

Continued from Part 4

Many Gentoo configurations, such as excluding rsync categories in /usr/portage, installing layman to manage overlays, setting up a local rsync mirror, or having Portage pull binary packages first all require editing configurations found in /etc/make.conf. While we can attempt to manage /etc/make.conf, we end up with the same issues with managed /etc/portage/package.use.

A hint at a better way can be found in the instructions for layman. There, the instructions says to add

source /usr/local/portage/layman

into /etc/make.conf. bash writers would instantly recognize this as the directive for sourcing another bash script. Ideally then, we would have

source /etc/portage/chef/make.conf

inside /etc/make.conf, then have Chef manage that make.conf file. That file in turn can source other files generated by the recipes, such as exclude_categories. In the case of exclude_categories, we need to append an option flag to PORTAGE_RSYNC_EXTRA_OPTS. The cascading inclusion would look like this:

/etc/make.conf
-> /etc/portage/chef/make.conf
  -> /etc/portage/chef/conf.d/rsync_exclude

Now, bash scripters may notice that the source directive is a bash directive, and so assume that we can do something like

source `ls /etc/portage/chef/conf.d/*`

... but sadly it doesn't work that way. (Besides, do we really want to automatically include everything in there?) We need a way for recipes to register a configuration, register it, and then have /etc/portage/chef/make.conf regenerated. This actually took a bit of doing:

1. I used a definition to create portage_conf definition. A recipe-writer should just know, they can add their overrides by using a definition file. In the case of exclude_categories, I used:
   

   portage_conf :rsync_excludes do
     appends [ :PORTAGE_RSYNC_EXTRA_OPTS, "--exclude-from=#{node[:gentoo][:rsync][:exclude_rsync_file]}" ]
   end

   
   The recipe writer is responsible for using valid flags. There is no validation done at compile-time.


2. My first attempt at registering the conf file was naïve. I attempted to declare a variable, portage_confs inside the portage recipe, and had the portage_conf definition append symbols to it. That did not work. I'm not entirely sure how everything works, but at the end of the day, portage_confs was not scoped, and so the DSL automatically tried to create a new resource.
   
   I next attempted to create a resource class so that the method_missing in the DSL would pick it up properly. However, since this is a fake resource that does not properly duck-type to a real resource, that failed miserably too.
   
   I ended up creating a custom container class for the sole purpose of pushing portage confs to.



3. While the container actually worked, it was not registering the portage conf properly. I had thought it had something to do with not notifying the resources properly, so I spent some time create an execute resource that would reset /etc/portage/chef/make.conf and then called the template to rebuild it. While that part worked, it still did not address the fact that the template was still pulling in an empty array.
   
   This part revealed to me a bit of how Chef works under the hood. This was hinted by the documentation for the ruby_block resource. Chef compiles all the resources and then runs it. ruby_block lets you run a block of code with the other resources. So of course, registering the portage conf would not work the way I had it because it statically declared what goes into /etc/portage/chef/make.conf ... which is an empty array.
   
   My solution? I broke apart the portage recipe into a second piece, make_conf recipe. The latter contains the template to make /etc/portage/chef/make.conf plus an execute resource that will reset the file. This works.

I don't think this is the ideal solution. Maybe there is something better for (2). I also don't like how I am rebuilding the file by deleting this. It does not take advantage of the backup mechanism that the template resource has, yet there is no :recreate action in template. (Maybe I should write one?). Feel free to comment here or on at Github if you think you have a better way of putting this together.

Update: Thanks to the guys at #chef @ irc.freenode.net I was able to clean up the code by reopening the resource. I didn't know you can still access the variables of a resource and manipulate it, but now it makes more sense.

Adventures with chef-server and Gentoo, part 4

Continued from Part 3

When I followed the instructions for chef-overlay, I noticed a feature for Portage that I never knew existed. For those not familiar with Gentoo, you can specify fine-grained control over the exact version number of compile-time features with four files, /etc/portage/package.use, /etc/portage/package.keywords, /etc/portage/package.unmask, and /etc/portage/package.mask.

The official documentation shows using these as files. However, the instructions from chef-overlay shows that you can make those files as directories, and put multiple files in those directories. For example, instead of:

/etc/portage/package.use

you can instead have,

/etc/portage/package.use/ruby
/etc/portage/package.use/chef
/etc/portage/package.use/java

I wish I had known about this years earlier. My old Gentoo laptop that I used for Rails development has been around since 2006, and has accrued a large set of use flags and keywords. (It is time to wipe that laptop clean and rebuild).

One cool thing about this setup is that autounmask will handle this layout gracefully. For example, if I were to autounmask =net-misc/rabbitmq-server-1.7.2-r2, it will create a file /etc/portage/package.keywords/autounmask-rabbitmq-server instead of attempting to inserting the change into the base /etc/portage/package.keywords ... which may have custom changes a human have made. Human-editable files would require the use of git to manage well. The files generated from autounmask can be added and removed atomically.

I'm not entirely sure this is still the way to go with a Gentoo + Chef infrastructure. Reading the paper, Why Order Matters: Turing Equivalence in Automated Systems Administration, one of the key things for congruent infrastructure is being able to rebuild at anytime. Does having it broken out like that make it easier? I think it is, but we will see.

I have pushed up a portage recipe that sets this up. I took some time to make sure that any legacy files gets backed up and moved into the new directory structure. Again, I'm not sure this is entirely a great idea since it tempts people to attempt to apply Chef, a congruence tool, on what is effectively a convergence problem. I'm leaving it there for now simply because the stem-cell images I am using had some of those files already in place (most notably, turning off the threads use flag for the ruby package).

Adventures with chef-server and Gentoo, part 3

Continuing from Part 2

The next step comes in a bundle of two. I am targeting the Rackspace Cloud platform. Each Rackspace slice gives you two ethernet devices. eth0 points to a public IP address and eth1 gives you a private IP address in the 172.* range. Packets going through eth0 incurs a charge while packets routed through eth1 does not. Since the default, non-nanite install of Chef assumes you will be polling chef-server every so often, having lots of these slices running around gets expensive if routing through the outside addresses.

So what we really want is a way to reference each of the nodes by the private IP address. There are several ways to do this. Since these are non-public addresses, you cannot use the Rackspace DNS server nor any of the public DNS services. 37Signals uses djbdns; others have suggested PowerDNS. The lowest-barrier-of-entry solution I have found so far queries the server index and builds a /etc/hosts. I chose this solution, with some modifications.

First, the /etc/hosts file emitted by the recipe described in that blog post maps the FQDN to 127.0.0.1. That's not what I want. If I want to use 127.0.0.1, I want to reference it with localhost and not the FQDN. Years ago, I remember reading how the Linux loopback device performs insanely faster than going through a regular ethernet device stack. I don't know if that is true now or where I can find the performance figures for it. Regardless, what I don't want is the ip reference for a FQDN changing on me. I have started a Rackspace cookbook here and dropped the ghetto DNS recipe in there along with my modifications.

Second, I do not fully agree with the Opscode design decision to use Fully-Qualified Domain Names. Philosophically, I'm adopting the Chef infrastructure as part of a deliberate strategy based on the principles of extropy. In short, stable forms arises from unstable substrates, and when the economics of the unstable substrate reaches a critical point, the substrate becomes disposable. They don't matter anymore to those who are operating at the higher abstraction. The fallout of that is that the unstable substrate supporting the stable abstraction becomes anonymous.

In practice, I want anonymous server VMs that can turn into anything, very much like pluripotent stem cells. I'd rather have the authoritative Chef node id be some sort of random md5 hash. If I wanted named servers, I'd do that on top of the anonymous nodes by using tags. I don't really want to manage which node is db0 and db1; by keeping these anonymous, I can create parameterized infrastructure. What we have planned at Sparkfly include an architectural building block called a "segment", which has a well-defined set of components spread across a group of two or three servers. Rather than specifying each node and each role for node, this should be all done programmatically using a higher-level description.

On the other hand, we will always have certain servers that must always be referenced. One would be chef-server. I would need a site-wide hosts table referencing that. (But even this can eventually be built on top of anonymous nodes).

Looking through the docs, there were some extra hoops I'd have to jump through to set the node name differently from that of the FQDN. For now, since I'm simply trying to get a working infrastructure up and running before I start digging in and trying to make more sweeping changes, I wrote a kludge that creates an anonymous name and appends that to the /etc/hosts. There is another supporting Gentoo script added into /etc/conf.d/local.start which detects the presence of a file, /etc/conf.d/bootstrap and runs kludge script. Run it, and you get something like node-1ee5ddee46d52c269b898bf216

What's next are rake tools to help query and reference based on tags. I also reserve the right to change my mind. Maybe I'll get tired of copying & pasting non-sensical md5 hashes when working with knife. After all, strategy is pragmatic and concerns what is, rather than its idealized form.