Cutting our build and deploy time by half using Rails, Docker and Elastic Beanstalk

Sat, Jan 21, 2017 4-minute read

My team has recently transitioned our infrastructure to using Docker and Elastic Beanstalk, a powerful duo. Docker, giving us the ability to create our environments in a predictable way which allows consistency across our test, staging and production environments. Elastic Beanstalk, providing a customizable PaaS that is stable and scalable and low-touch once it has been properly configured.

It has proven to be a great solution, but it can take a long time building our Docker images from scratch each time we need to do a test build or a deploy in our CI/CD process.

Here’s what we were experiencing:

Like I said, this process takes a long time when you compare it to our existing CI/CD process which leverages an existing environment. But it’s nice to build a Docker image from scratch each time. There’s something satisfying in knowing that we’re whipping up a fresh container to run our tests or to deploy the latest version of our application. It’s clean. It’s reproducible.

So how can we have the best of both worlds: build a brand new Docker image each time, but reduce the amount of time it takes to build this Docker image?

To come up with a solution, we looked at Docker image inheritance and Rails fragment caching.

Cache the “fragment”

Docker images are engineered to inherit from one another. The first command in your Dockerfile is “FROM.” Let’s look at the Docker image inheritance for our application. scratch > ubuntu:14.04 > ourapp:1.0

What if we added an additional “hop” so that we could cache a few of the steps in our build process? scratch > ubuntu:14.04 > ourapp:cached > ourapp:1.0

When building ourapp:cached, we will perform the more costly operations that are the most stable parts of our build process:

• Downloading the ubuntu:14.04 base layer
• Installing the required packages
• Copy .ruby-version, Gemfile and Gemfile.lock from the current application
• Installing RVM, Ruby and the required gems

When building ourapp:1.0, we will use ourapp:cached as the base image and perform version-specific operations on top of it:

• Copy the current application into /webapp/current
• Reinstall the required gems (This handles the case where a gem has been added and removed since the cached image was built. This process takes seconds opposed to minutes since the majority of the gems have already been downloaded and installed.)
• Precompile the assets
• Define our entrypoint

Expire the cache

We aren’t uploading our cached layers to Docker Hub or ECR because a) that process also takes time and b) our building of ourapp:cached is isolated to our CI/CD server.

To ensure that the gemset in our cached image don’t get completely out of sync with the current version of our application, we use a cronjob to periodically purge ourapp:cached. # crontab 0 1 1 * * sh ~/scripts/remove_cached_docker_images.sh``# ~/scripts/remove_cached_docker_images.sh echo "Removing ‘cached’ images..." sudo docker rmi ourapp:cached

Summary and Extra Tips

This process has proven to be very effective for us in expediting the amount of time it takes to build a Docker image for a test build or for deploying a new application version.

In addition to this solution, we’ve done a few other things to improve the efficiency of our CI/CD process and are always looking for ways to make further improvements.

• Using Elastic Beanstalk “artifacts”

• Elastic Beanstalk was not built with Docker in mind. In fact, it currently shoe-horns in EC2 Container Service “under the covers”. That being said, certain awsebcli commands aren’t optimized for Docker.

• For example, when you run eb deploy, it actually zips up your entire application directory and transfers it to S3. It doesn’t need to do this! For Elastic Beanstalk with Docker, the application version only needs to contain the Dockerrun.aws.json file and potentially your .ebextensions directory. To prevent eb deploy from uploading your entire application directory, consider using artifacts.# .elasticbeanstalk/config.yml deploy: artifact: application.zip``# Makefile (or something similar) zip application.zip Dockerrun.aws.json .ebextensions/* git add application.zip eb deploy ourapp --staged

• Deploying to our web and worker environments asynchronously

• We split our build process to create a web and worker environment-specific Docker images from a common base layer. We allow our CI/CD server to start the deploy process to each respective environment asynchronously.

• One gotcha: if your web and worker environments exist within the same Elastic Beanstalk application, there is a potential for your application versions to collide and your worker Docker image to mistakenly deploy to your web environment. To prevent this from happening, consider using version_labels with the eb deploy command.

About the Author

Aaron Plantenga is a Senior Software Engineer working on the Guest Services squad at the XO Group since 2015. When he’s not in front of a computer, he enjoys running and hiking, Malaysian food and horror movies.Originally published at blog.eng.xogrp.com.