Monitoring BackgrounDRb workers with God

2008-08-14T12:04:00.014-05:00

Updated 2008/09/24 for latest version of backgroundrb 1.0.4

The other day on our staging server, I noticed that the BackgrounDRb queue worker had died. As it turned out, the queue worker had died over 3 months ago!!

There was no cause for alarm as the staging server isn't critical but this did start me to worrying. We needed to implement a monitoring solution which not only verified that BackgrounDRb was running but also that particular workers were running.

As we had just implemented god monitoring with a custom condition for another issue, its a slam dunk to do the same again. (Thanks to Jesse Newland and his god tutorial at AtlRUG.)

Here's the configuration file that got it done for us.


  1 #run on command line with 'god -c backgroundrb.god -D'
  2 RAILS_ROOT = '/var/www/rails/rollbook/current'
  3 
  4 #load required rails and backgroundrb files
  5 require File.dirname(__FILE__) + '/../boot'
  6 require File.dirname(__FILE__) + '/../environment'
  7 require 'erb'
  8 $LOAD_PATH << "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib"
  9 require "#{RAILS_ROOT}/vendor/plugins/backgroundrb/lib/backgroundrb.rb"
 10 
 11 #create custom condition for checking that QueryProcessingWorker is running
 12 MiddleMan = BackgrounDRb::ClusterConnection.new
 13 module God
 14   module Conditions
 15     class Backgroundrb < PollCondition
 16       def initialize; super; end
 17       def valid?; true; end
 18 
 19       def test
 20         begin
 21           queue_worker = MiddleMan.all_worker_info.values.flatten.select { |w| :queue_processing_worker == w[:worker] }
 22           queue_worker.empty?
 23         rescue #if all_worker_info raises exception, then bdrb isn't running and we were unable to connect
 24           true
 25         end
 26       end
 27     end
 28   end
 29 end
 30 
 31 God.watch do |w|
 32   w.name      = 'backgroundrb'
 33   w.interval  = 1.minute
 34   w.restart   = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop && #{RAILS_ROOT}/script/backgroundrb -e production start"
 35   w.stop      = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production stop"
 36   w.start     = "cd #{RAILS_ROOT} && #{RAILS_ROOT}/script/backgroundrb -e production start"
 37   w.grace     = 1.minute
 38   w.pid_file  = "#{RAILS_ROOT}/tmp/pids/backgroundrb_11000.pid"
 39 
 40   w.start_if do |start|
 41     start.condition(:process_running) do |c|
 42       c.running  = false
 43     end
 44   end
 45 
 46   w.restart_if do |restart|
 47     restart.condition(:backgroundrb) do |c|
 48       #just restart it
 49     end
 50   end
 51 end

In the select call on line 21, you can modify the condition to access :job_key or :status as well. Obviously, you need to modify RAILS_ROOT for your situation.

If you have suggestions for improvement or questions, hit me up in the comments. Enjoy!

Ack, a better grep for coders in 3 easy steps

2007-12-28T15:58:00.000-05:00

Ack has this nice colorized output when searching code files. Some may say that grep can do the same thing (it can), but there's no need for arg'ing it up with ack. There's a similar ruby version, rak, but I've heard it isn't quite as fast as ack. Ack works well enough for me and the install process isn't that painful.

There's two ways to install listed over at ack; I chose the CPAN route (since I never know when I'll need to install some more perl goodies).

How to install ack on Mac OS X leopard in 3 easy steps.

install mac ports
sudo port install perl5.8
sudo cpan -i App::Ack

Two Minute Spates

Monitoring BackgrounDRb workers with God

Ack, a better grep for coders in 3 easy steps