LocalyticsEng

Stager: Staging Environments on Autopilot

Brian Zeligson -

At Localytics we take our code live early and often. It helps us to test features that are in development, and communicate progress throughout the company. We do this to work out in the open, catch issues and deviations as quickly as possible, and just to feel more connected to the end results of our work. Easy deployment is a good thing.

The way we used to test our changes on community staging environments was pretty complicated. It required a manual process of deployment using a Capistrano task, on a limited number of reserved boxes. You'd drop into chat to ask if anyone was using the server you planned to deploy to, count to 20 (or something) and pull the trigger. If you were to object to this, and you happened to be getting a cup of coffee during that countdown, your features were clobbered by the next deployment.

As the team grew, we knew that our process around staging had to improve to keep our team efficient and open. Enter Stager.

Stager takes a Github pull request, and turns it into an isolated running instance of that codebase, entirely automatically. There is no more process around staging your work, it just happens as a byproduct of your normal workflow of shipping code.

One week after integrating, Engineering could not imagine working without it. Fast forward 6 months to present, and we've since integrated it further into our systems and our product, services, support, and even finance teams are just as accustomed to having links to Stager instances available for working with forks of our platform.

Stager is solving a problem for us that is not unique to Localytics, so we decided early on that we would open source Stager, and paid close attention to flexibility and configurability.

Customizations and third party integrations are super to easy to add and the included plugins provide solid templates to start with, so it is easy to tailor to your specific environment and workflow.

The remainder of this post will walk you through setting up a Stager instance that responds to Github pull requests, exactly as we're using it at Localytics. If you're interested in setting up a Stager server to integrate with your own environment, read on.

And remember, if you need further customization beyond what is covered here, it's covered in detail in the documentation.

Pre-requisites and initial set up

Stager is based on Sinatra, Docker, and Nginx, so you need those in place first. For the walkthrough, I've launched a new Amazon EC2 running Ubuntu 14.04. After running the commands under the Prerequisites and Quick Setup section of the README, I can now visit http://demo.stager.io and see that things are running correctly.

Note that Stager will use subdomains for routing to staged instances, so you also need a wildcard dns entry for the domain name you are using to point to your Stager server.

Dockerizing the application

This is a subject that is covered well elsewhere, and there are several methods available. To keep things simple here, we're going to use the manual "change and commit" process outlined on the Docker website.

Our application is going to be a simple static website, helping us move quickly and focus on getting our Stager instance integrated. We'll run the website using a ruby one-liner, so let's start with an image that already has ruby installed. The Trusted Build Ruby image seems like a good way to do this, so let's run the following:

docker pull dockerfile/ruby

Once that's finished, start a container based on the image as follows:

docker run -i -t dockerfile/ruby /bin/bash

We can now clone our project into the image. For this walkthrough, let's use the stager demo project here, courtesy of html5up.net. Fork the repo then run the following

cd /
git clone https://github.com/YOUR-FORK/stager-demo.git

Next we need a boot script. This is what will kick off the web server when Stager launches a container. Run the following inside the container:

echo 'ruby -run -ehttpd /stager-demo/ -p 3000' > /run.sh
chmod +x /run.sh

Last we need to save our changes to a new image. In a new terminal, with the container still running, run the following:

docker ps
# docker commit [CONTAINER ID FROM docker ps] stager-demo
docker commit 439b stager-demo

Once that's done, you can exit the container session in your original terminal. We now have a docker image for running our app, and we can move on to configuring Stager.

Configuring Stager

Stager needs a minimum configuration before it knows what to do with your docker image. Edit the config.yml in the stager directory, and change it to look like the following:

 images:
  stager-demo:
    port: 3000
    command: '/bin/bash /run.sh'
    container_create_params:
      AttachStdin: true
      AttachStdout: true
      AttachStderr: true
      OpenStdin: true
      StdinOnce: true
      Tty: true
routing_strategy: 'NginxRoutingStrategy'
nginx:
  - target_dir: '/etc/nginx/sites-enabled'
  - template_path: './request_handlers/nginx.conf.erb'
authentication_strategy: 'BasicAuthentication'
users:
  - 'testuser:testpassword'

What we are doing here is telling Stager how to use the Docker image we have created, how to route requests from the host machine to a running container, and how we will authenticate. We are also defining a user. Note the BasicAuthentication strategy uses plain-text password stored in the config, and is really only intended for demonstration, or closed network implementation. When we integrate with Github in a future step, this will get better.

We've done enough now to use Stager for managing our instances. Let's test with curl from localhost.

If you have been running Stager this whole time, kill and restart it.

On Stager server:

ps -ax | grep 'thin'
kill -9 [process ID from previous command output]
thin start -d

On localhost:

curl --user testuser:testpassword -d 'image_name=stager-demo&container_name=test' http://demo.stager.io/launch

Stager will respond with a url. Specifically it adds a url-friendly "slugged" version of your container_name parameter as a subdomain on top of your Stager host. Our staged application is now available there: http://test.demo.stager.io

Github integration

Configuration

Stager uses request handlers to allow customization and extension of the normal workflow via configuration file changes. To turn on Github integration we need to add fields to our config.yml file:

images:
  stager-demo:
    port: 3000
    command: '/bin/bash /run.sh'
    container_create_params:
      AttachStdin: true
      AttachStdout: true
      AttachStderr: true
      OpenStdin: true
      StdinOnce: true
      Tty: true
routing_strategy: 'NginxRoutingStrategy'
nginx:
  - target_dir: '/etc/nginx/sites-enabled'
  - template_path: './request_handlers/nginx.conf.erb'
authentication_strategy: 'BasicAuthentication'
event_listeners:
  - 'LaunchOnGithubPullRequestOpened'
  - 'KillOnGithubPullRequestClosed'
post_launch_handlers:
  - 'AddGithubPullRequestCommentOnLaunch'
github:
  incoming_auth:
    user: 'basic_auth_user_for_git_hook'
    password: 'basic_auth_password_for_git_hook'
    hook_secret: 'hook_secret'
  outgoing_auth:
    user: 'username_for_outgoing_github_requests'
    password: 'password_for_outgoing_github_requests'

Briefly let's look at what each new section does.

event_listeners/post_launch_handlers: These are different classes of request handlers, which are configurable and pluggable classes that allow modification of the normal behavior of Stager. Here we have added the behavior to launch and kill staged instances when a PR is opened and closed, comment on a PR when a staged instance is launched for it, and provide an endpoint to assist in taking us through the Github OAuth flow, particularly for use with the Stager CLI gem.

github/incoming_auth:

We will need to add a webhook to Github in order for Stager to stay in sync with our PRs, and this section tells Stager what credentials to expect with those postbacks to ensure the request is coming from Github.

To create the postback, click Settings on your repository, then Webhooks & Services.


On the Webhooks page, click Add Webhook, and fill out the form on the following screen.

In Payload URL, put the following, replacing the creds with your own and the domain with the domain where your Stager server resides:

http://user:password@demo.stager.io/event_receiver

Note the user:password you use before the actual URL, then update user and password under github:

  • incoming_auth: in your Stager config.yml to match
  • Content type should be application/json
  • Enter a secret in the Secret field, and update the hook_secret: key under github:incoming_auth: in your config.yml to match.

Under "Which events would you like to trigger this webhook?"

  • click "Let me select individual events"
  • deselect "Push"
  • select "Pull Request"

Example:

With this, the webhook is configured, and Stager is able to authenticate any requests from Github, ensuring these calls cannot be faked by anyone else.

github/outgoing_auth:

In order for Stager to leave a comment on a Pull Request after it has launched an environment, it needs to be able to authenticate to Github.

In the case of a private repository, this only requires read access to the repository, so it is recommended that you create a separate Github account with minimum required privileges to your repository for this purpose.

For public repositories, it is still best practice to have a separate Github account solely for use by Stager, since you will be storing the credentials plain text in the Stager config file.

Once you have created and/or chosen the account under which Stager will post to Github, add the username and password to the user: and password: fields under the github: outgoing_auth: sections of your Stager config.yml

We now have completed configuration for Github integration. Restart your stager instance for the config changes to take effect.

ps -ax | grep 'thin'
kill -9 [process ID from previous command output]
thin start -d

Boot Script

Now Stager will launch a new instance whenever we open a Pull Request, our docker image needs to know what to do with the information it receives. Stager passes the post body parameters from the webhook to a new container as environment variables, so we can infer the repository and branch from the $image_name and $container_name environment variables respectively.

Let's open another shell session with our stager-demo image and update the /run.sh script from Step #2 as follows:

docker run -i -t stager-demo /bin/bash
vim /run.sh

Update the startup script to look as follows:

cd /stager-demo
git fetch
git checkout $container_name
git pull origin $container_name
ruby -run -ehttpd . -p 3000

Note if you are working with a private repository, your image needs to have an ssh-identity added for a user with access to the repo, along with necessary keys. This can be the same user you created for the PR comments above, with read-only access.

Again leave this session running, and in a separate shell, run the following

docker ps
# docker commit [CONTAINER ID] stager-demo
docker commit 5a23 stager-demo

You can now exit the shell session from the first console with those changes saved to the docker image.

At this point our Github integration is done, and we can test it out by opening a pull request.



You should also notice that when you close or merge the Pull Request, the instance disappears, and when changes are pushed to the branch that has an open PR, the instance refreshes itself.

Github Authentication

Configuration

Right now we are controlling access for direct requests to Stager with a plain-text user/password list in config.yml, which is way less than ideal. Since we are already integrated with Github, it makes sense to use Github to authenticate any direct requests made to Stager. We can do this by changing our authentication strategy to GithubAuthentication, which infers a repo name from the image_name parameter on any request, and uses a Github OAuth token to allow a request if it comes from someone with push access to that repo.

Let's update our config.yml to use this authentication strategy. Change your config.yml to look like the following:

 images:
  stager-demo:
    repo_owner: account_name_which_owns_repo_inferred_from_image_name
    port: 3000
    command: '/bin/bash /run.sh'
    container_create_params:
      AttachStdin: true
      AttachStdout: true
      AttachStderr: true
      OpenStdin: true
      StdinOnce: true
      Tty: true
routing_strategy: 'NginxRoutingStrategy'
nginx:
  - target_dir: '/etc/nginx/sites-enabled'
  - template_path: './request_handlers/nginx.conf.erb'
authentication_strategy: 'BasicAuthentication'
event_listeners:
  - 'LaunchOnGithubPullRequestOpened'
  - 'KillOnGithubPullRequestClosed'
  - 'GithubAuthorization'
post_launch_handlers:
  - 'AddGithubPullRequestCommentOnLaunch'
github:
  client_id: 'github_app_client_id_for_creating_oauth_token'
  client_secret: 'github_app_client_secret_for_creating_oauth_token'
  incoming_auth:
    user: 'basic_auth_user_for_git_hook'
    password: 'basic_auth_password_for_git_hook'
    hook_secret: 'hook_secret'
  outgoing_auth:
    user: 'username_for_outgoing_github_requests'
    password: 'password_for_outgoing_github_requests'

Briefly let's look at the config we have added:

repo_owner: This is how the authentication strategy will know which repository to check for access. If you forked the demo application, this will be your Github username.

GithubAuthorization: This creates an endpoint that performs the Github web OAuth flow, so that you can use Stager to retrieve an OAuth token. This is used by the Stager CLI to make things more convenient, detailed below.

client_id:/client_secret: This is used by the GithubAuthorization endpoint, to perform the web OAuth flow. To get this information, register a new Github Application for Stager to use. Be sure to put the following in the "Authorization Callback URL" field.

http://your-stager-domain.com/event_receiver

The client_id and client_secret are displayed on the following page.

Stager CLI

Stager is now configured to use Github for authenticating direct requests. Unfortunately, this means that all of your calls to Stager need to be signed with a Github token, so in order to make this easier, we've put together the Stager CLI gem.

Rather than curling around manually retrieved Github tokens, the CLI works with Stager to take you through the web OAuth flow and save your token in a config, providing you a higher level syntax for working with Stager that will add your authentication information to all requests under the hood (In case you are thinking of alternate integrations, it is equally extensible as the main Stager project).

To get set up with the Stager CLI, follow the installation instructions in the README and then run the following your localhost:

stager configure endpoint=http://your-stager-domain.com auth_strategy=Github

Now we can launch and kill stager instances with the following syntax:

stager launch(/kill) IMAGE_NAME CONTAINER_NAME

Remembering that our demo app is now reading branch name from the CONTAINER_NAME parameter, let's launch the master branch. Notice that the first time you run this it will take you through the web OAuth flow to retrieve a Github token. This will be saved in a local config file so you do not have to re-auth on future requests.

stager launch stager-demo master

After some prompts, you should eventually see a browser opened to master.your-stager-host.com, running the master branch of your application.

With that, you now have a running example of how we've been staging our work at Localytics for the past few months, and should have a clear idea of how you can make this work for your project (hint: it's mostly about the docker image.)

A word of caution for public repositories: Be aware that you are essentially granting anyone with a Github account access to execute any code they want on your Stager server by simply opening a Pull Request.

At Localytics we are only using this for private repositories, but if you want to use it for a public repo your security concerns become a much bigger challenge. Remember that the code is running inside a Docker container, and you control access to what application runs (inside the run.sh boot script.) Beyond that, it is open season.

Be careful and mindful, and if you do decide to take on the challenge of securing a public repo integrated with Stager, please share your experience and approach, in a gist, on your blog, or in the comments.