A powerful way to deploy to Heroku
11:34:13. CTO: Do you deploy?
11:34:37. Dev 1: I do.
11:36:15. CTO: Still deploying?
11:36:21. Dev 1: Yup.
11:37:13. CTO: Deployed?
11:37:16. Dev 1: Wait, it’s restarting.
11:37:22. CTO: …Restarted?
11:37:25. Dev 1: Yeah, tell Max he can have a look.
11:40:01. CTO: Max says that nothing has changed. Are you sure it’s deployed?
11:40:48. Dev 1: I AM SURE! Works for me. But wait, what the hell…
11:41:03. Dev 2: Oh, guys, are you deploying? I’ve just deployed my branch.
11:41:09. Dev 1: …!
11:41:14. CTO: …!
I am not making this up. Back in 2011 we had such a dialogue in Campfire.
What to say? A process involving many people should be apparent for everyone. That’s why airports provide flight information on big screens, not restricting it to a flight dispatcher’s terminal!
If we already are here, in this chat, let’s also deploy here! And let all stages and the result of deployment be obvious. This story tells how I implemented such a scheme.
Here’s how it works in Shuttlerock’s Slack channel:
This solution might be useful for you too!
We tend to use Heroku in our projects. Just as any real thing, Heroku has its pros and contras. It’s more expensive than a virtual server at, say, DigitalOcean. But an ability to sleep at night, while Heroku DevOps engineers are awake, comes with a price.
Heroku makes the deployment process simple: just a
git push. After some wait
the app restarts and works; at least in theory. There are some practical nuances, though.
Nuance 1. Assets
That’s right: CSS, JS & friends. If you don’t do anything special about them, your
Heroku app will work. But assets will be served from Ruby:
your expensive web dynos will pose as Apache. First, it’s slow (real
Apache and nginx are much, much faster). Second, while Rack code is pushing the compiled
application.js into a network buffer, incoming requests sit in line and wait.
Such a scheme is apparently only suitable for personal (as in “I’m the only user”)
Therefore most real sites use a CDN for serving assets. For example, assets can be uploaded to Amazon S3 which interoperates with Amazon CloudFront, the Amazon’s CDN service. Things are similar with Rackspace and other cloud platforms. Nothing is better for static files than a CDN – take this as an axiom.
It must be noted that Rails app needs a manifest file to be able to generate
proper asset URLs. This file is generated during
In Rails 4 it’s in
It’s the manifest that allows app to generate that long hexadecimal suffix
when you specify
application-3dda9c3...yaddayadda.js file has been uploaded to the CDN,
the generated link would work.
But how does it play with Heroku? By default Heroku runs
rake assets:precompile during deploy process. This command populates
directory with compiled asset files, including the manifest. But how could those files make
their way to the CDN? How can we be sure that the manifest is 100% actual?
There’s an asset_sync gem for that.
It adds a rake task which runs after
rake assets:precompile and uploads
everything to S3. As such, the synchronization works directly from a Heroku build server
Unfortunately this solution is not perfect. It slows down the deploy process, especially in large projects. In general, asset precompiler is smart. It caches its results and wouldn’t regenerate things that didn’t change. Look at this local experiment:
The second run was a no-op (and thus fast!). But this trick doesn’t work with Heroku, because build server always starts from scratch. Full precompilation and S3 sync will take place every time, even if there were no asset changes. And it hurts!
Nuance 2. Migrations
If you use an SQL database, you need to run migrations from time to time. Heroku
doesn’t automate this, you have to execute
heroku run rake db:migrate by youself.
However, you can run this command only after deployment, when appropriate
db/migrate directory are in place. There’s a possibility that
just deployed Ruby code refers to a non-existent database attribute (the appropriate migration hasn’t
been run yet), and an unlucky user gets “500 Internal Error”.
To mitigate this problem Heroku suggests to go into maintenance mode before deploying. The full deployment script will look like this:
heroku restart is called for the database scheme to be reloaded. In production mode it’s loaded
only once, during boot; if there’s no
age attribute in
User#age= methods won’t be created. Even
after the migration runs any code calling these methods will still generate an exception.
Unfortunately, even this protective script doesn’t make you 100% safe. There’s
still a chance that some unlucky user visits your site between
heroku restart. He’s still going to get his 500 error. Not to be solved!
Now let’s imagine that it takes several minutes for assets to be compiled and synced. Not surprising for a large app. And we’re in maintenance mode during this time!..
A path to desired system
So far we’ve described the scope of the problem:
- Obviousness of deployment process and its results.
- Assets compilation and uploading to the CDN.
Having spent some time thinking, I came to the following solution:
- The deployment process runs inside a Jenkins instance.
- Assets are precompiled and uploaded to S3 before Heroku deployment. If there
were no changes since last deploy (as told by Git history), nothing is done.
If something has changed,
rake assets:precompilecan use local cache to speed up the precompilation process.
- Deployment is triggered by Hubot, a bot which sits in the chat and interacts with Jenkins. It also reports deploy status.
- The migration run is optional. Two deploy modes are introduced: regular (following the full script mentioned above) and quick, when migrations are not run at all.
The whole system consists of the following parts:
- A hubot instance.
- A custom Hubot script which implements
deploycommand and interacts with Jenkins.
- A Jenkins instance with properly configured jobs.
bin/deploy.shscript in the app source tree. This script is run by Jenkins during deployment.
- A Ruby module inside the app which implements S3 upload.
Let’s discuss some of these parts.
Hubot is easily deployed to Heroku.
Jenkins is a Java monster. Not my favorite type of apps, but it’s very stable and there’s a wide selection of plugins. Writing your own plugin is not as easy as with Hubot, but there’s no need. So, what can Jenkins give us?
- Git support. Jenkins can clone and update the repo, and checkout any branch you want.
- Build history with logs. If something fails, the log is available to all engineers cool enough to log in to Jenkins.
- Job queue. If two deploy commands were given, the second one would wait until first one finishes. If the second one is a mistake, you can quickly log in to Jenkins and cancel it.
- Parallel builds. If you have several apps or several versions of a single app (staging, production), deploys can proceed in parallel.
In essence, Jenkins is a fancy shell to run bash scripts working on Git checkouts.
Now let’s set up and configure every part. We’ll assume that our Heroku app is named coolapp.
- GIT plugin.
- Build Authorization Token Root Plugin. Allows deploy jobs to be triggered by an incoming HTTP request.
- Notification plugin. Will notify Hubot about build process.
I also install Green Balls for appearance, ChuckNorris Plugin for fun, Matrix Authorization Strategy Plugin for convenient access control, Credentials Plugin for managing additional private keys and SCM Sync Configuration Plugin to mirror Jenkins config changes into a Git repo.
The UNIX user under which Jenkins daemon runs (usually just
should have all needed versions of Ruby installed. Another ansible role,
zzet.rbenv, proved itself very useful
in this respect.
Now let’s move on to creating a deploy job. Let’s give it a meaningful name:
Every app version (staging, production, …) will need a separate job. Then configure notification:
The URL field must contain the full Hubot URL with
Further down the page, check the “Parameterized Build” checkbox. The parameters we are going to create will be available to a deploy script as environment variables. We’ll also be able to pass their values from outside upon triggering a build with HTTP request. The following parameters should be added:
||Heroku application name|
||Git branch Name|
|QUICK||Boolean||Off||Quick mode doesn’t run migrations|
Select Git in “Source Code Management”, provide a URL of your repo and
origin/$BRANCH into the branch field:
This will allow us to deploy any branch present in the repo.
Now specify a remote token in “Build Triggers” and uncheck other checkboxes:
What’s left is adding a single build step (“Execute Shell”):
So, we’re setting everything up and delegating actual deployment to
Note that if you’re creating the second, the third, etc. job, don’t start from scratch, use copy mode instead:
Now let’s move on to Hubot.
Install Hubot following the instruction. By the way, pick a catchy and fun name for your bot instead of “hubot”! E.g. in Shuttlerock we have rodney.
scripts directory. This version works well for Slack; if you use other adapters,
the script might need minimal changes.
Then you’ll have to edit a single line in the
Specify all apps/app versions which you are going to deploy here. E.g. this line could look like this:
This assumes two versions of the main app and two versions of the monitoring app.
Other configuration is done by setting environment variables for the Hubot instance. Let’s look at what’s needed.
HUBOT_JENKINS_URLvariable must contain the root Jenkins URL (e.g.,
HUBOT_JENKINS_BUILD_TOKENcontains the token which you put into the job configuration (
0W5CT73cFV4ia89N9Sa87S644v3twA9Pin our example). It’s assumed that this token is the same for all apps and app versions.
Now set 4 variables for each app from
HUBOT_PRODUCTION_APP. The name of Heroku app.
coolapp-stagingin our case.
HUBOT_PRODUCTION_DEFAULT_BRANCH. The default branch to be deployed (when name is not specified). E.g.
HUBOT_PRODUCTION_JOB. The Jenkins job name.
HUBOT_PRODUCTION_ACL. This variable manages access to deployment. It should either contain the list of E-mails of admitted people or be set to
everyone. Note that you can see the full E-mail list with
hubot show userscommand.
Variable names are obtained from the app name. E.g. monitoring staging
At this point if everything has been configured the right way, a bot command (
hubot deploy to staging)
will trigger a Jenkins build with proper parameters. Now let’s look at
the deployment core,
The script is provided in the same gist; here we are going to examine how it works, function by function.
This shows how a bot command’s “tail” is used: you can pass additional
parameters in there.
hubot deploy to staging is the basic story, but
hubot deploy to staging and recompile assets and clear cache does more:
influence the process (see further).
public/assets/CURRENT_SHA file contains SHA1 of the latest Git commit
vendor/assets (if you have other asset folders,
add them as additional
git log arguments).
If nothing has changed since then, no compilation would occur. In other case
precompilation and S3 upload are triggered (
rm -rf public/assets is done just in case, it’s
For S3 upload to work, add lib/tasks/cloud_assets.rake
to your source tree. For latter module to work you’ll also need the
fog_aws gem; don’t
forget to add it to your
The FogCloudAssets module does an incremental S3 upload.
recompile assets mode, as we already saw, triggers full recompilation,
reupload assets does a full S3 upload: all files are reuploaded even if they
are already on S3.
cleanup assets removes all assets on S3 which are missing
from the current manifest.
save_deploy_information and commit
We manage assets before deployment, but how can we make sure that the correct
manifest is present in the deployed code? I solved this problem in a following way:
a temporary branch is created on every deploy; the manifest file is added and committed
to Git; then
git push --force is run. Heroku sees the committed manifest
and skips the
rake assets:precompile step.
It’s not very elegant, but brings new possibilities! The
function, for example, generates a
lib/deploy_info.rb file which has the following format:
This makes what’s currently deployed apparent. We use ActiveAdmin:
The panel looks quite nice and informative:
For it to work locally, add the following
lib/deploy_info.rb to your source tree:
After a tour through details, let’s consider the main deployment logic.
bin/deploy.sh: main logic
QUICK param, we follow either short or long path. Other things
are also shown here:
skip herokumode allows to precompile and upload assets, but skip the Heroku push. Rarely used, but sometimes helpful.
- We have
Rails.cache.clearin Shuttlerock. The commented out part shows how this can be used. Full deploy proactively runs
db:migrate, while quick mode allows you to tell
... and clear cacheto the bot.
Following this logic you can easily add your own deploy flags and modes.
Neither Hubot, nor Jenkins is to be touched; you’ll only have to handle
The deploy command instructions
Whew, you’ve set everything up. What can you do now?
hubot deploy to production deploys the production app (default branch).
hubot quick deploy feature/something-really-cool to staging deploys the specified branch (
feature/…) to staging
in quick mode; no migrations are run.
hubot deploy to production and recompile assets deploys the default branch to production,
forcefully recompiling assets even if there was no change.
hubot disable deploys to staging temporary disables staging deploys (useful
if you do maintenance or are debugging something and don’t want
anybody to interfere). A reason can be specified:
hubot disable deploys to staging because it hurts.
hubot enable deploys to staging reenables deploys.
Note that the enable/disable feature works best if you add a Redis “brain” to your Hubot: see the hubot-redis-brain plugin.
What we have now is a classy modular deployment system. Its advantages for Rails:
- Convenient, manageable Heroku deploys.
- Asset compilation is a separate step, not slowing down the Heroku push. No asset changes – no recompilation/sync!
But the system is not specifically tied to Rails. It also has general advantages for all technologies:
- Everyone sees what’s happening.
- There’s a history of successful and failed deploys.
- Everyone who’s allowed can deploy. Including novices and testers who have hard time with setting up and updating local dependencies (Heroku Toolbelt, Ruby, bundler, node.js, etc.)
- Any Git branch can be deployed.
- Any technology can be supported by writing a proper script.
My colleagues tell me that it’s the best deployment system ever. I trust their word 😄. Why don’t you try?