# A powerful way to deploy to Heroku

### Prologue

11:34:13. CTO: Do you deploy?

11:34:37. Dev 1: I do.

11:36:15. CTO: Still deploying?

11:36:21. Dev 1: Yup.

11:37:13. CTO: Deployed?

11:37:16. Dev 1: Wait, it’s restarting.

11:37:22. CTO: …Restarted?

11:37:25. Dev 1: Yeah, tell Max he can have a look.

11:40:01. CTO: Max says that nothing has changed. Are you sure it’s deployed?

11:40:48. Dev 1: I AM SURE! Works for me. But wait, what the hell…

11:41:03. Dev 2: Oh, guys, are you deploying? I’ve just deployed my branch.

11:41:09. Dev 1: …!

11:41:14. CTO: …!

I am not making this up. Back in 2011 we had such a dialogue in Campfire.

What to say? A process involving many people should be apparent for everyone. That’s why airports provide flight information on big screens, not restricting it to a flight dispatcher’s terminal!

If we already are here, in this chat, let’s also deploy here! And let all stages and the result of deployment be obvious. This story tells how I implemented such a scheme.

Here’s how it works in Shuttlerock’s Slack channel:

This solution might be useful for you too!

### Heroku

We tend to use Heroku in our projects. Just as any real thing, Heroku has its pros and contras. It’s more expensive than a virtual server at, say, DigitalOcean. But an ability to sleep at night, while Heroku DevOps engineers are awake, comes with a price.

Heroku makes the deployment process simple: just a git push. After some wait the app restarts and works; at least in theory. There are some practical nuances, though.

#### Nuance 1. Assets

That’s right: CSS, JS & friends. If you don’t do anything special about them, your Heroku app will work. But assets will be served from Ruby: your expensive web dynos will pose as Apache. First, it’s slow (real Apache and nginx are much, much faster). Second, while Rack code is pushing the compiled version of application.js into a network buffer, incoming requests sit in line and wait. Such a scheme is apparently only suitable for personal (as in “I’m the only user”) sites.

Therefore most real sites use a CDN for serving assets. For example, assets can be uploaded to Amazon S3 which interoperates with Amazon CloudFront, the Amazon’s CDN service. Things are similar with Rackspace and other cloud platforms. Nothing is better for static files than a CDN – take this as an axiom.

It must be noted that Rails app needs a manifest file to be able to generate proper asset URLs. This file is generated during rake assets:precompile. In Rails 4 it’s in public/assets directory:

It’s the manifest that allows app to generate that long hexadecimal suffix when you specify <%= javascript_include_tag :application %> in your layout. If this application-3dda9c3...yaddayadda.js file has been uploaded to the CDN, the generated link would work.

But how does it play with Heroku? By default Heroku runs rake assets:precompile during deploy process. This command populates public/assets directory with compiled asset files, including the manifest. But how could those files make their way to the CDN? How can we be sure that the manifest is 100% actual?

There’s an asset_sync gem for that. It adds a rake task which runs after rake assets:precompile and uploads everything to S3. As such, the synchronization works directly from a Heroku build server during deploy.

Unfortunately this solution is not perfect. It slows down the deploy process, especially in large projects. In general, asset precompiler is smart. It caches its results and wouldn’t regenerate things that didn’t change. Look at this local experiment:

The second run was a no-op (and thus fast!). But this trick doesn’t work with Heroku, because build server always starts from scratch. Full precompilation and S3 sync will take place every time, even if there were no asset changes. And it hurts!

#### Nuance 2. Migrations

If you use an SQL database, you need to run migrations from time to time. Heroku doesn’t automate this, you have to execute heroku run rake db:migrate by youself. However, you can run this command only after deployment, when appropriate files in db/migrate directory are in place. There’s a possibility that just deployed Ruby code refers to a non-existent database attribute (the appropriate migration hasn’t been run yet), and an unlucky user gets “500 Internal Error”.

To mitigate this problem Heroku suggests to go into maintenance mode before deploying. The full deployment script will look like this:

heroku restart is called for the database scheme to be reloaded. In production mode it’s loaded only once, during boot; if there’s no age attribute in users table, User#age and User#age= methods won’t be created. Even after the migration runs any code calling these methods will still generate an exception.

Unfortunately, even this protective script doesn’t make you 100% safe. There’s still a chance that some unlucky user visits your site between heroku maintenance:off and heroku restart. He’s still going to get his 500 error. Not to be solved!

Now let’s imagine that it takes several minutes for assets to be compiled and synced. Not surprising for a large app. And we’re in maintenance mode during this time!..

### A path to desired system

So far we’ve described the scope of the problem:

1. Obviousness of deployment process and its results.
3. Migrations.

Having spent some time thinking, I came to the following solution:

• The deployment process runs inside a Jenkins instance.
• Assets are precompiled and uploaded to S3 before Heroku deployment. If there were no changes since last deploy (as told by Git history), nothing is done. If something has changed, rake assets:precompile can use local cache to speed up the precompilation process.
• Deployment is triggered by Hubot, a bot which sits in the chat and interacts with Jenkins. It also reports deploy status.
• The migration run is optional. Two deploy modes are introduced: regular (following the full script mentioned above) and quick, when migrations are not run at all.

The whole system consists of the following parts:

1. A hubot instance.
2. A custom Hubot script which implements deploy command and interacts with Jenkins.
3. A Jenkins instance with properly configured jobs.
4. A bin/deploy.sh script in the app source tree. This script is run by Jenkins during deployment.
5. A Ruby module inside the app which implements S3 upload.

Let’s discuss some of these parts.

#### Hubot

I fell in love with Hubot when I first saw it. I use it all the time now. My favorite commands are image and excuse.

Hubot provides a convenient API to your Slack (HipChat, Campfire) chat. Custom command handlers are relatively easy to implement: it’s just Javascript, after all.

Hubot is easily deployed to Heroku.

#### Jenkins

Jenkins is a Java monster. Not my favorite type of apps, but it’s very stable and there’s a wide selection of plugins. Writing your own plugin is not as easy as with Hubot, but there’s no need. So, what can Jenkins give us?

• Git support. Jenkins can clone and update the repo, and checkout any branch you want.
• Build history with logs. If something fails, the log is available to all engineers cool enough to log in to Jenkins.
• Job queue. If two deploy commands were given, the second one would wait until first one finishes. If the second one is a mistake, you can quickly log in to Jenkins and cancel it.
• Parallel builds. If you have several apps or several versions of a single app (staging, production), deploys can proceed in parallel.

In essence, Jenkins is a fancy shell to run bash scripts working on Git checkouts.

### Configuration

Now let’s set up and configure every part. We’ll assume that our Heroku app is named coolapp.

#### Jenkins

Jenkins is easy to set up. In Shuttlerock I used ansible with Stouts.jenkins role, but you can just install the package. We’ll need the following Jenkins plugins:

I also install Green Balls for appearance, ChuckNorris Plugin for fun, Matrix Authorization Strategy Plugin for convenient access control, Credentials Plugin for managing additional private keys and SCM Sync Configuration Plugin to mirror Jenkins config changes into a Git repo.

The UNIX user under which Jenkins daemon runs (usually just jenkins) should have all needed versions of Ruby installed. Another ansible role, zzet.rbenv, proved itself very useful in this respect.

Now let’s move on to creating a deploy job. Let’s give it a meaningful name:

Every app version (staging, production, …) will need a separate job. Then configure notification:

The URL field must contain the full Hubot URL with /hubot/jenkins_status path, e.g. https://my-hubot-instance.herokuapp.com/hubot/jenkins_status.

Further down the page, check the “Parameterized Build” checkbox. The parameters we are going to create will be available to a deploy script as environment variables. We’ll also be able to pass their values from outside upon triggering a build with HTTP request. The following parameters should be added:

Name Type Default Value Description
APP String coolapp-staging Heroku application name
BRANCH String master Git branch Name
DEPLOYER String (empty) Deployer name
QUICK Boolean Off Quick mode doesn’t run migrations
EXTRA String (empty) Extra parameters

Select Git in “Source Code Management”, provide a URL of your repo and put origin/\$BRANCH into the branch field:

This will allow us to deploy any branch present in the repo.

Now specify a remote token in “Build Triggers” and uncheck other checkboxes:

What’s left is adding a single build step (“Execute Shell”):

So, we’re setting everything up and delegating actual deployment to the bin/deploy.sh script.

Note that if you’re creating the second, the third, etc. job, don’t start from scratch, use copy mode instead:

Now let’s move on to Hubot.

#### Configuring Hubot

Install Hubot following the instruction. By the way, pick a catchy and fun name for your bot instead of “hubot”! E.g. in Shuttlerock we have rodney.

Add deploy.coffee to the scripts directory. This version works well for Slack; if you use other adapters, the script might need minimal changes.

Then you’ll have to edit a single line in the deploy.coffee file:

Specify all apps/app versions which you are going to deploy here. E.g. this line could look like this:

This assumes two versions of the main app and two versions of the monitoring app.

Other configuration is done by setting environment variables for the Hubot instance. Let’s look at what’s needed.

1. The HUBOT_JENKINS_URL variable must contain the root Jenkins URL (e.g., https://jenkins.example.com).

2. The HUBOT_JENKINS_BUILD_TOKEN contains the token which you put into the job configuration (0W5CT73cFV4ia89N9Sa87S644v3twA9P in our example). It’s assumed that this token is the same for all apps and app versions.

3. Now set 4 variables for each app from APPS:

• HUBOT_PRODUCTION_APP. The name of Heroku app. coolapp-staging in our case.
• HUBOT_PRODUCTION_DEFAULT_BRANCH. The default branch to be deployed (when name is not specified). E.g. master.
• HUBOT_PRODUCTION_JOB. The Jenkins job name. coolapp-staging-deploy for us.
• HUBOT_PRODUCTION_ACL. This variable manages access to deployment. It should either contain the list of E-mails of admitted people or be set to everyone. Note that you can see the full E-mail list with hubot show users command.

Variable names are obtained from the app name. E.g. monitoring staging has HUBOT_MONITORING_STAGING_APP, HUBOT_MONITORING_STAGING_DEFAULT_BRANCH etc.

At this point if everything has been configured the right way, a bot command (hubot deploy to staging) will trigger a Jenkins build with proper parameters. Now let’s look at the deployment core, bin/deploy.sh.

#### bin/deploy.sh

The script is provided in the same gist; here we are going to examine how it works, function by function.

set_extra_flags

This shows how a bot command’s “tail” is used: you can pass additional parameters in there. hubot deploy to staging is the basic story, but hubot deploy to staging and recompile assets and clear cache does more: CLOUD_ASSETS_RECOMPILE and CLEAR_CACHE variables influence the process (see further).

compile_assets

The public/assets/CURRENT_SHA file contains SHA1 of the latest Git commit that changed app/assets or vendor/assets (if you have other asset folders, add them as additional git log arguments).

If nothing has changed since then, no compilation would occur. In other case precompilation and S3 upload are triggered (rm -rf public/assets is done just in case, it’s not required).

For S3 upload to work, add lib/tasks/cloud_assets.rake and lib/fog_cloud_assets.rb. to your source tree. For latter module to work you’ll also need the fog_aws gem; don’t forget to add it to your Gemfile:

The FogCloudAssets module does an incremental S3 upload. If recompile assets mode, as we already saw, triggers full recompilation, reupload assets does a full S3 upload: all files are reuploaded even if they are already on S3. cleanup assets removes all assets on S3 which are missing from the current manifest.

save_deploy_information and commit

We manage assets before deployment, but how can we make sure that the correct manifest is present in the deployed code? I solved this problem in a following way: a temporary branch is created on every deploy; the manifest file is added and committed to Git; then git push --force is run. Heroku sees the committed manifest and skips the rake assets:precompile step.

It’s not very elegant, but brings new possibilities! The save_deploy_information function, for example, generates a lib/deploy_info.rb file which has the following format:

This makes what’s currently deployed apparent. We use ActiveAdmin:

The panel looks quite nice and informative:

For it to work locally, add the following lib/deploy_info.rb to your source tree:

After a tour through details, let’s consider the main deployment logic.

bin/deploy.sh: main logic

Depending on QUICK param, we follow either short or long path. Other things are also shown here:

• skip heroku mode allows to precompile and upload assets, but skip the Heroku push. Rarely used, but sometimes helpful.
• We have rake cache:clear call Rails.cache.clear in Shuttlerock. The commented out part shows how this can be used. Full deploy proactively runs cache:clear along with db:migrate, while quick mode allows you to tell ... and clear cache to the bot.

Following this logic you can easily add your own deploy flags and modes. Neither Hubot, nor Jenkins is to be touched; you’ll only have to handle stuff in bin/deploy.sh.

### The deploy command instructions

Whew, you’ve set everything up. What can you do now?

hubot deploy to production deploys the production app (default branch).

hubot quick deploy feature/something-really-cool to staging deploys the specified branch (feature/…) to staging in quick mode; no migrations are run.

hubot deploy to production and recompile assets deploys the default branch to production, forcefully recompiling assets even if there was no change.

hubot disable deploys to staging temporary disables staging deploys (useful if you do maintenance or are debugging something and don’t want anybody to interfere). A reason can be specified: hubot disable deploys to staging because it hurts.

hubot enable deploys to staging reenables deploys.

Note that the enable/disable feature works best if you add a Redis “brain” to your Hubot: see the hubot-redis-brain plugin.

### Conclusion

What we have now is a classy modular deployment system. Its advantages for Rails:

• Convenient, manageable Heroku deploys.
• Asset compilation is a separate step, not slowing down the Heroku push. No asset changes – no recompilation/sync!

But the system is not specifically tied to Rails. It also has general advantages for all technologies:

• Everyone sees what’s happening.
• There’s a history of successful and failed deploys.
• Everyone who’s allowed can deploy. Including novices and testers who have hard time with setting up and updating local dependencies (Heroku Toolbelt, Ruby, bundler, node.js, etc.)
• Any Git branch can be deployed.
• Any technology can be supported by writing a proper script.

In Shuttlerock all active projects are deployed using this system. E.g. we wrote a script for an angular.js app using bower and gulp.

My colleagues tell me that it’s the best deployment system ever. I trust their word 😄. Why don’t you try?