The important part is in your
Host heroku.com ForwardAgent yes
After that it's no big deal. Just use an ssh URL to Github (or wherever,
the Github part doesn't really matter) in your
Gemfile and off you go!
source :rubygems ruby '1.9.3' gem 'sinatra', :git => 'email@example.com:darkhelmet/sinatra.git' gem 'thin' gem 'heroku'
So if you need a private gem on Heroku for work or something, go right ahead.
It's no big deal.