Creating an Automated Feature Test Suite with Capybara/Selenium, Docker & AWS
Creating an Automated Feature Test Suite with Capybara/Selenium, Docker & AWS
Feature tests are important. They simulate the actual functionality of a product, mimicking what a user will click-through and interact with across your website. Working on a project that doesn’t feature test comes at a cost. Uncaught bugs could unknowingly interrupt your user’s experience on key areas of your product. Not only can this cost members turning to other services, but in turn costs revenue for your company.
Since these tests have many steps to them, running them takes time. Most developers don’t have this kind of time to walk away from their desks for 10 or more minutes at a time while they wait for all features to be tested.
One solution to this problem was to create an automated feature test suite, outside of the unit testing suite. This would run every hour on a regular basis, with reporting for the team.
Getting Started with Capybara/Selenium
Starting the project was easy enough; I began to write feature tests. Feature (or functional) testing is testing the complete functionality of an application. These typically test a complete ‘flow’ or area of a website, such as signing up for an account. On the other end, unit tests typically test individual methods and smaller pieces of code. These are typically quick to run and large in number. For writing feature tests, I used Capybara and Selenium Webdriver.
Our feature tests need to run on a website, in a real browser, unlike unit tests that only need access to your local source code. Selenium ‘drives’ an actual browser, letting you watch the flow of a feature test in real time. Upon learning more about the capabilities of this webdriver, I wrote the rest of the critical feature test cases. While headless browsers like PhantomJS do exist and have the reputation of running more quickly, I wanted the ability to actually watch the tests run for an easier debugging experience. It was beneficial to be able to incorporate Pry while targeting my selectors. With Selenium Webdriver, you get the experience of watching the browser in real time, if you choose to. Additionally, since this is to run on a schedule, speed was not my top priority, but rather transparency.
Incorporating Docker
Once I had a driver navigating the places we needed to assert functionality, the next step was to learn about Docker. Docker describes itself as a ‘container service,’ which to me means wrapping up your code into a neat, self-contained package, and shipping it somewhere that it can very easily be replicated and run again as you intend it to. For testing, this is an attractive choice to eliminate the variable nature of environment setups from one computer to the next. When you run a program from inside a Docker container, you’re ensuring that it’s going to run in isolation from everything else on your system, and that all of its dependencies exist just inside of the container.
This presents some interesting challenges in itself though. I started to realize that applications I took for granted during my local development environment needed to be explicitly installed in my Docker container. Most importantly, I needed a browser to be installed in order to run Selenium Webdriver tests. These directions of what needs to be built for the app to run smoothly go inside a Dockerfile at the root of your project.
Setting it to a Schedule
Once I had the tests running in a Docker container, I needed to think about getting it to run itself on a schedule. It’s simply not convenient for someone to take 10 minutes out of every hour to run this manually. So, I explored a variety of scheduling tools. There are gems such as Sidekiq, Resque, and Whenever, that let you create ‘jobs’ of a sort that recur whenever you define them to. These all had their own benefits, but in the end, it came down to wanting a simpler solution that integrates more seamlessly with Docker. I didn’t need a separate gem or scheduling system, but rather just a script that runs the test suite and then sleeps for an hour or so. While it was a fun exploration of all these helpful tools, choosing to not go with a gem reduced bloat for my Docker container.
At this point, I had a feature test suite that ran on a recurring basis through Docker. Once all of the tests finished running each cycle, I needed a way to alert the team of any important results. I formatted out the results to an HTML file with a dashboard view that then gets uploaded to an S3 bucket. Additionally, any screenshots of a test that had failed were uploaded too, in case further investigation is required. These are taken automatically through the super helpful gem, capybara-screenshot. I also installed the Slack-notifier gem, and set up a webhook to ping the team upon test completion.
The final step for this project is to push it up to Amazon Web Services, so that it’s not running on just my machine. While it is packaged up through Docker, I still was running it locally on my server and leaving it up all day for the schedule to take place.
We deployed this app on a docker swarm cluster, which is a tool that supports running many docker containers on one AWS EC2 instance. This is ultimately a more economical way to run this project and avoids running several instances, each with their own container, that could ultimately be under-utilized. In conjunction with Shipyard, a tool built on docker-swarm for managing docker resources, I can now run make commands whenever I have a change to the app, and push the updated build in order to continuously deploy.
Summary and Benefits
At this point, it runs every hour, in the cloud! Some results of this suite are the benefits it provides to each new deployment. Being able to assure critical functionality on an ongoing basis can help catch any new bugs quickly, and with this automation, it doesn’t need to be prompted from the developer’s side. This confidence increases productivity, and supports the initiative of continuous delivery.
ABOUT THE AUTHOR
Jackie Feminella
Jackie is a Software Engineering Intern working on the Guest Services squad at the XO Group since June 2016. She enjoys making monthly playlists on Spotify, rock climbing, coffee, and yoga.Originally published at blog.eng.xogrp.com.