Making big jobs easy: clustering with Ruby and ssh the easy way

This is a blog posting about my experiences so far trying to make a simplistic clustering solution that does the job I require of it.

I have quite a large task which can be subdivided up into many smaller similar tasks.  In a way, one could implement the map and reduce methods of Google and Starfish, but that is overkill for the time being.

My platform of preference is Ruby at the moment, due to the speed at which one can develop a new solution.  Ruby is obviously not the fastest language around, and technically one should be embarrassed about raising a clustered application in Ruby, but it does have the ability to have inline C code for the speed sensitive areas.

So, there are several clustering solutions available including DRB, Rinda (similar to Java's Linda), some form of XML RPC, SOA, SOAP and others.

I have gone for the simplest.

I simply open a connection, using the standard Ruby process pipes, to SSH.  I secure-copy (scp/rsync) the Ruby script over to the cluster clients, and then execute it through my Ruby script from the master.  Using the Ruby process pipes I send the data to be dealt with to the input and I simply wait as many minutes as necessary for the output to be received on STDOUT.  I then save off the results and do it all again.

I have hacked Rake to allow thread pools.  Normally Rake will create as many threads as there are multiple synchronous tasks.  In my case, this could be many thousands.  So, using the thread pool Ruby code available on the net I have limited it to eight threads with a defined server for each to ensure distribution of load.

This has made a long and arduous task very easy, and because the requirements of the cluster processing platform are so limited (secure shell and Ruby) it is very easy to add a new machine to the processing group.

There is also the potential in the future to use Amazon's elastic computing (EC2) service where one could instantiate 100 virtual machines to do the hard work in a very short amount of time.  It would be very easy to extend this simple clustering strategy to that, although one would certainly see definite improvements through using some form of client/server objects a la DRB.


Technorati Tags: cluster, ssh, ruby, distributed, drb, rinda, linda