Wednesday 27 June 2007

Making big jobs easy: clustering with Ruby and ssh the easy way

This is a blog posting about my experiences so far trying to make a simplistic clustering solution that does the job I require of it.

I have quite a large task which can be subdivided up into many smaller similar tasks.  In a way, one could implement the map and reduce methods of Google and Starfish, but that is overkill for the time being.

My platform of preference is Ruby at the moment, due to the speed at which one can develop a new solution.  Ruby is obviously not the fastest language around, and technically one should be embarrassed about raising a clustered application in Ruby, but it does have the ability to have inline C code for the speed sensitive areas.

So, there are several clustering solutions available including DRB, Rinda (similar to Java's Linda), some form of XML RPC, SOA, SOAP and others.

I have gone for the simplest.

I simply open a connection, using the standard Ruby process pipes, to SSH.  I secure-copy (scp/rsync) the Ruby script over to the cluster clients, and then execute it through my Ruby script from the master.  Using the Ruby process pipes I send the data to be dealt with to the input and I simply wait as many minutes as necessary for the output to be received on STDOUT.  I then save off the results and do it all again.

I have hacked Rake to allow thread pools.  Normally Rake will create as many threads as there are multiple synchronous tasks.  In my case, this could be many thousands.  So, using the thread pool Ruby code available on the net I have limited it to eight threads with a defined server for each to ensure distribution of load.

This has made a long and arduous task very easy, and because the requirements of the cluster processing platform are so limited (secure shell and Ruby) it is very easy to add a new machine to the processing group.

There is also the potential in the future to use Amazon's elastic computing (EC2) service where one could instantiate 100 virtual machines to do the hard work in a very short amount of time.  It would be very easy to extend this simple clustering strategy to that, although one would certainly see definite improvements through using some form of client/server objects a la DRB.

Technorati Tags: , , , , , ,


Anonymous said...

ninest123 16.03
gucci outlet, louis vuitton, louboutin outlet, chanel handbags, ray ban sunglasses, louis vuitton, replica watches, longchamp outlet, michael kors outlet, jordan shoes, ugg boots, tory burch outlet, oakley sunglasses, louis vuitton outlet, oakley sunglasses, polo ralph lauren outlet, michael kors outlet, burberry outlet online, nike air max, michael kors outlet, nike outlet, tiffany and co, michael kors outlet, ray ban sunglasses, ray ban sunglasses, louis vuitton, michael kors outlet, louboutin, burberry, ugg boots, louboutin shoes, polo ralph lauren outlet, tiffany jewelry, ugg boots, replica watches, longchamp outlet, oakley sunglasses, christian louboutin outlet, prada outlet, oakley sunglasses, michael kors, louis vuitton outlet, nike free, cheap oakley sunglasses, uggs on sale, longchamp, nike air max, prada handbags, ugg boots

Anonymous said...

nike free, mulberry, ray ban pas cher, new balance pas cher, hogan, converse pas cher, hollister, ralph lauren pas cher, coach outlet, michael kors, lacoste pas cher, true religion jeans, sac longchamp, north face, michael kors, hermes, nike trainers, true religion jeans, nike roshe, burberry, timberland, nike roshe run, nike blazer, true religion outlet, nike air max, vanessa bruno, louboutin pas cher, hollister pas cher, nike free run uk, longchamp pas cher, replica handbags, air max, lululemon, oakley pas cher, michael kors, coach outlet, ralph lauren uk, true religion jeans, nike air max, vans pas cher, tn pas cher, north face, sac guess, air jordan pas cher, coach purses, michael kors, ray ban uk, nike air max, air force, abercrombie and fitch, nike huarache

Anonymous said...

ugg boots uk, ugg,ugg australia,ugg italia, moncler, ugg,uggs,uggs canada, moncler, moncler, louis vuitton, canada goose outlet, hollister, louis vuitton, moncler, barbour, moncler, sac louis vuitton pas cher, ugg pas cher, karen millen, louis vuitton, lancel, bottes ugg, canada goose, canada goose uk, gucci, replica watches, juicy couture outlet, wedding dresses, coach outlet, montre pas cher, pandora charms, louis vuitton, converse outlet, moncler, links of london, barbour jackets, thomas sabo, toms shoes, canada goose, pandora jewelry, swarovski, pandora charms, marc jacobs, canada goose, moncler outlet, supra shoes, juicy couture outlet, doudoune canada goose, moncler, swarovski crystal, canada goose outlet, canada goose, ray ban, pandora jewelry
ninest123 16.03

Unknown said...

oakley sunglasses
michael kors uk
adidas stan smith
red bottoms
versace shoes
hermes handbags
mont blanc pens
jordan pas cher
true religion outlet
pandora outlet

Anonymous said...

replica bags louis vuitton hermes fake d6g01e5v34 replica goyard bags have a peek at this website s6x49i8v03 high quality designer replica replica bags high quality discover this info here w0d19i1s94 replica louis vuitton bag replica bags louis vuitton