Let's get some background information out of the way.
I'm working on a new application, and am using crazy new things that I haven't had a chance to really use before. Rails 3, MongoDB, Redis and Resque, HTML5, etc. 1
With all these things, I figured I'll just use EngineYard since I can pretty much do whatever I want with the server, and they have a lot of the "stuff" taken care of. But they don't support MongoDB out of the box, and you have to some magic with chef scripts, and then to keep your Resque workers running, you need to write stuff for that too, and so on.
I did that originally, but frankly, I'd rather be working on my app than the infrastructure.
Heroku to the rescue
As you might know, I'm a big fan of Heroku. This blog runs on it, and I don't have to ever think about it. It. Just. Works.™
Now, there's nothing wrong with EngineYard, but I want to deal with as little infrastructure crap (or at least the boring parts) as possible, and Heroku lets me do that. Plus, I get some things for free, like NewRelic RPM.
redistogo
Along comes redistogo with their Heroku integration (for beta users, which I am) and a blog post on how to use Resque with Heroku. What, what, what? You can just alias the rake task and Heroku's system doesn't know the difference. Oh Em Gee.
My wife has been quite adamant about the money2
This app I'm working on, I'm hopefully going to have to pay for at some point, as I hope enough people will want to use it that the free stuff from Heroku just won't cut it. However, the less I can pay the better, and background job workers aren't free on Heroku. They are, fortunately, billed by the second.
So let's get some auto-scale up in this bitch shall we?3
First off, grab my fork of resque. I added after_enqueue hook support, which is needed for the auto-scaling.4
Update: Chris Wanstrath pulled in my changes, so any version of resque 1.10 or higher has after_enqueue support.
You can use this in a Gemfile like so:
| gem 'resque', '>= 1.10.0' | |
| gem 'heroku' # You will need the heroku gem for this too. |
Now throw this in your lib directory,
| require 'heroku' | |
| module HerokuResqueAutoScale | |
| module Scaler | |
| class << self | |
| @@heroku = Heroku::Client.new(ENV['HEROKU_USER'], ENV['HEROKU_PASS']) | |
| def workers | |
| @@heroku.info(ENV['HEROKU_APP'])[:workers].to_i | |
| # For Cedar | |
| # @@heroku.ps(ENV['HEROKU_APP']).count { |a| a["process"] =~ /worker/ } | |
| end | |
| def workers=(qty) | |
| @@heroku.set_workers(ENV['HEROKU_APP'], qty) | |
| # If you're running on the Cedar stack, do this instead. | |
| # @@heroku.ps_scale(ENV['HEROKU_APP'], :type=>'worker', :qty=>qty) | |
| end | |
| def job_count | |
| Resque.info[:pending].to_i | |
| end | |
| end | |
| end | |
| def after_perform_scale_down(*args) | |
| # Nothing fancy, just shut everything down if we have no jobs | |
| Scaler.workers = 0 if Scaler.job_count.zero? | |
| end | |
| def after_enqueue_scale_up(*args) | |
| [ | |
| { | |
| :workers => 1, # This many workers | |
| :job_count => 1 # For this many jobs or more, until the next level | |
| }, | |
| { | |
| :workers => 2, | |
| :job_count => 15 | |
| }, | |
| { | |
| :workers => 3, | |
| :job_count => 25 | |
| }, | |
| { | |
| :workers => 4, | |
| :job_count => 40 | |
| }, | |
| { | |
| :workers => 5, | |
| :job_count => 60 | |
| } | |
| ].reverse_each do |scale_info| | |
| # Run backwards so it gets set to the highest value first | |
| # Otherwise if there were 70 jobs, it would get set to 1, then 2, then 3, etc | |
| # If we have a job count greater than or equal to the job limit for this scale info | |
| if Scaler.job_count >= scale_info[:job_count] | |
| # Set the number of workers unless they are already set to a level we want. Don't scale down here! | |
| if Scaler.workers <= scale_info[:workers] | |
| Scaler.workers = scale_info[:workers] | |
| end | |
| break # We've set or ensured that the worker count is high enough | |
| end | |
| end | |
| end | |
| end |
…and extend any Resque job classes with it.
| class ScalingJob | |
| extend HerokuResqueAutoScale | |
| def self.perform | |
| # Do something long running | |
| end | |
| end |
You'll need to set some Heroku config variables for your application name, username (email), and password. You can also of course alter the scaling logic to do whatever you need it to do. Mine scales up workers after a job is enqueued based on the number of pending jobs, and after a job finishes, turns off the workers if there are no more jobs pending.
Make sure to set the workers to 1 on your command line just to make sure it works, since if you haven't done anything before that requires payment, Heroku might require you to confirm, in which case the auto-scaling fails.
Now, your workers only run when they need to. You don't need to have a bunch of workers running for those times the job queue does get a little backed up, and you don't even need to keep track of it, because it will scale itself without you having to mess with it. Enjoy.
- Let's see how many other buzzwords I can cram into this post…
- Homer pays off the mob to get rid of competition for Marge's pretzel business, and Fat Tony comes back claiming "[his] wife has been quite adamant about the money" in The Twisted World of Marge Simpson
- By auto-scaling, I mean spin up workers when we have work to do, and shut them down when there are no jobs.
- I sent a pull request. If anything changes, I'll update this post.