December 03, 2015

Implementing a Centralized Directory Service for AWS Infrastructure with Amazon Simple AD and SSSD

By Matt Juszczak

Localytics Engineering has doubled in size in the past year. The number of SSH keys floating around on running instances was absolutely unwieldy.

We responded with a centralized directory service for our AWS infrastructure. We used open tools in a straightforward way, and our team loves it.

The following is a step-by-step walkthrough of our solution based on Amazon services (plus new features in Ubuntu and RHEL/CentOS that made our lives a little bit easier).

We'll cover the reasoning behind our decisions along with instructions on how you can set this up yourself.

Bonus: below are links to our open source Chef cookbooks (which will perform most of the work for you)

Let's get started!

LDAP

First, we needed an LDAP-compatible backend directory. We were hesitant to use OpenLDAP because of the effort required to customize the community Chef cookbook for our needs. Also, we prefer to utilize "as-a-service" tools whenever possible to simplify administration.

Enter Amazon's Simple AD. Although geared to replace Microsoft Active Directory (which means it’s targeted for Windows instances), a Simple AD instance is still LDAP (actually, Samba) under the hood. It was our best choice, albeit with a few limitations.

We were aware of the directory integration features of PAM and NSS with pamldap and nssldap, respectively, along with nslcd. Historically these modules have been troublesome (and not well documented). To make matters worse, their caching support is nonexistent, so an LDAP connectivity outage would leave servers inaccessible.

System Security Services Daemon (SSSD)

Enter SSSD, the centralized access point for all authentication and authorization requests for pam, nss, sudo, and more. SSSD is powerful, as it:

You must be using at least Ubuntu 14.04 LTS and/or CentOS 6.x, as they ship with a recent SSSD version that includes these key features.

While SSSD provides a mechanism for fetching SSH keys from LDAP, OpenSSH still needs to read and trust those keys as if they were in the usual location (.ssh/authorized_keys).

As of CentOS 6.x and Ubuntu 14.04 we no longer have to redeploy configuration management or run complicated scripts just to replace SSH keys! This is achieved through an AuthorizedKeysCommand configuration parameter, which specifies a command string to return keys for a given user.

The SSSD service allows us to centralize password authentication, public/private key authentication, host access, and sudo roles and access.

Amazon Simple AD Setup

Configuring Simple AD is probably the most challenging part of this setup. In brief, we’ll cover:

To begin, spin up a Simple AD instance. You'll need to load some custom schema files. Create sudo.ldif with these contents, ssh.ldif with these contents, and sudoers.ldif with contents similar to this, which will create a root sudo role and gives all users access to it. Make sure to replace the Base DN references in each entry with the Base DN of your directory server (i.e., dc=example, dc=com).

To load these LDIFS, execute these commands in your terminal, being sure to replace placeholders with proper values:

$ ldbadd -H "ldap://example.com" sudo.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"
$ ldbadd -H "ldap://example.com" ssh.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"
$ ldbadd -H "ldap://example.com" sudoers.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"

If you get an error that --user is an invalid option, make sure you have the samba package installed alongside ldb-tools.

Now create a user so you can test your setup. In this example, the user is tuser. Once the user is created, it must be imported by issuing the associated ldbadd command:

$ ldbadd -H "ldap://example.com" users.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"

If you have a host with samba-tool installed, you have the option to set a password for the new account:

$ samba-tool user setpassword --newpassword "<password>" -H "ldap://example.com" --user "<Admin Account Username>" --password "<Admin Account Password>" tuser

You must also use samba-tool to grant authenticated users read access to the CN=Sudoers container that was created via previous LDIFs. These commands must be run against each AD Server (as this will not replicate):

$ samba-tool dsacl set -H "ldap://198.51.100.10" --user "<Admin Account Username>" --password "<Admin Account Password>" --objectdn="CN=Sudoers,DC=example,DC=com" --sddl="(A;CI;GR;;;AU)"
$ samba-tool dsacl set -H "ldap://203.0.113.20" --user "<Admin Account Username>" --password "<Admin Account Password>" --objectdn="CN=Sudoers,DC=example,DC=com" --sddl="(A;CI;GR;;;AU)"

Now add an SSH key to the test user you've created with something like this and apply the LDIF:

$ ldbmodify -H "ldap://example.com" add-key.ldif --user "<Admin Account Username>" --password "<Admin Account Password>"

Your Simple AD instance should now be properly configured. The next step is to configure SSSD and OpenSSH using a test instance and tie it into the directory service.

SSSD & OpenSSH Setup

First, spin up a vanilla Ubuntu 14.04 LTS instance, switch to a root shell, and edit /etc/resolv.conf to include your search domain and nameservers, making sure to use the correct IP addresses and domain name:

search example.com
nameserver 198.51.100.10
nameserver 203.0.113.20

Once that is in place, execute realm list in the CLI to test connectivity between your instance and Simple AD. You should see information about your Simple AD domain.

Assuming that everything is working so far, the next step is to install the sssd and samba packages and join the Simple AD domain:

$ apt-get -y update
$ apt-get -y install sssd sssd-ad sssd-ad-common sssd-tools realmd samba-common samba-libs samba-common-bin
$ realm join -U <Admin Account Username> example.com

There is a known bug that can cause a package error during realm join which we have worked around by temporarily using adcli in our Chef cookbook.

Your server should now be successfully joined to the domain. Unfortunately, realm doesn't give you a completely usable SSSD configuration file, so it's necessary to overwrite /etc/sssd/sssd.conf with something like this, being sure to replace placeholders with proper values.

Finally, add the following lines to /etc/ssh/sshd_config, which will tell OpenSSH to look for SSH keys via SSSD:

AuthorizedKeysCommand /usr/bin/sss_ssh_authorizedkeys
AuthorizedKeysCommand root

Be sure to execute service sssd restart and service ssh restart.

Testing Your Setup

To test your setup, run id tuser to verify that SSSD can talk to Simple AD and retrieve user and group information about your test user. Next, verify that it can access and return SSH keys with /usr/bin/sss_ssh_authorizedkeys tuser.

Finally, verify that your test user has root sudo privileges with /usr/bin/sudo -U tuser -l.

Chef Cookbook

Testing the configuration of Simple AD, SSSD, and OpenSSH is simple with our Chef cookbooks which are available on GitHub:

While you probably want to use an organization-specific wrapper cookbook, we have tried to make it as easy as possible to clone the SSSD cookbook and simply "get it working".

As noted above, our Chef cookbook temporarily uses adcli instead of realm to join the domain due to this bug. Future versions will revert back to using realm. To get started, make sure you have the chef-dk installed and a sane Ruby setup, then simply clone and set up the sssd cookbook and run the default test suite:

$ git clone git@github.com:localytics/chef-sssd
$ cd chef-sssd
$ chef gem install knife-solo_data_bag
$ chef exec kitchen test

In order to verify that it is possible to join and interact with your newly configured domain, copy .kitchen.local.yml.EXAMPLE to .kitchen.local.yml and update it to match your environment details, such as EC2 instance information, IP addresses of your primary and secondary AD servers, AD domain, and a user that has at least one public key and one sudo role configured.

The user that you specify is used by integration tests to verify that all services are functioning correctly. Our default test user, “Guest,” likely does not have the configuration necessary to pass the integration tests so you'll want to make this is “tuser” if you used our examples above.

Next, create an encrypted data bag key used for locally created data bags and configure an encrypted data bag with a valid administrative username and password, used for joining the domain with realm:

$ openssl rand -base64 512 | tr -d '\r\n' > test/integration/with-registration/encrypted_data_bag_secret_key
$ knife solo data bag create sssd_credentials realm -c .chef/solo.rb

The format of the data bag is:

{
  "id": "realm",
  "username": "administrator",
  "password": "password"
}

Once you're all set, simply chef exec kitchen test with-registration and voilà! If all tests pass, you've successfully configured your Simple AD server.

Comments and feedback on our setup are welcome!

Hero image above is of power lines and supporting structure in lane west of Main Street at Pender Street. March 10, 1914. Photo: British Columbia Electric Railway Company, CoV Archives, AM54-S4-: LGN 1241.