Deploying pam_ussh: A Comprehensive Guide

Introduction

As part of my travels and working with a large number of distributed linux systems we often need a better way to handle AAA(authentication, authorization and accounting), this wont help with 'accounting' but having a centralised machanisim to handle user sudo rights without requiring a 'phone home' approach like LDAP is really appealing.. enter Uber's Sudo PAM module - pam_ussh, a custom Pluggable Authentication Module (PAM) written in Go.

Below we'll go through the steps to compile the module, set up the necessary PAM configuration, and test the deployment.

What is pam_ussh?

pam_ussh is a PAM module designed to enhance SSH authentication by leveraging SSH certificates. The module checks if the connecting user's SSH public key is signed by a trusted Certificate Authority (CA) and optionally verifies that the certificate includes specific authorized principals. This can be particularly useful for organizations that need to enforce strict access controls.

Compiling pam_ussh

To get started with pam_ussh, you'll first need to compile the module. Below is the Dockerfile used to compile the module:

 1# Start from the official Ubuntu image
 2FROM ubuntu:latest
 3
 4# Set the working directory
 5WORKDIR /go/src/app
 6
 7# Install required packages
 8RUN apt-get update && apt-get install -y \
 9    golang \
10    ca-certificates \
11    make \
12    libpam0g-dev
13
14# Copy the current directory contents into the container at /go/src/app
15COPY . /go/src/app
16
17# Download Go modules
18RUN go mod tidy
19
20# Build the Go application
21RUN make

After building the Docker image, you can compile the module with the following commands:

1docker run --rm -it -v $(pwd):/go/src/app -w /go/src/app -e CGO_ENABLED=1 ubuntu:latest make
2sudo cp pam_ussh.so /etc/ssh/pam_ussh.so

This process compiles the pam_ussh module and copies the resulting shared object file to the appropriate directory for PAM modules.

Configuring PAM with pam_ussh

Once pam_ussh is compiled and placed in the appropriate directory, you'll need to configure PAM to use it. Here is a basic PAM configuration example:

1vagrant@node-1:~/pam-ussh$ cat /etc/pam.d/pam_test 
2#%PAM-1.0
3auth [success=1 default=ignore] /etc/ssh/pam_ussh.so ca_file=/etc/ssh/ca-users.pub
4  auth requisite                  pam_deny.so
5  auth required                   pam_permit.so

This configuration tells PAM to use the pam_ussh module for authentication, where the ca_file parameter points to the CA public key file.

If you need to require specific principals for authentication, you can modify the configuration as follows:

1vagrant@node-1:~/pam-ussh$ cat /etc/pam.d/pam_test 
2#%PAM-1.0
3auth sufficient /etc/ssh/pam_ussh.so ca_file=/etc/ssh/ca-users.pub authorized_principals=domain-admins,global-sudo,superusers,test-123 revoked_keys_file=/etc/ssh/revoked-keys
4
5  auth requisite                  pam_deny.so
6  auth required                   pam_permit.so

Here, the authorized_principals parameter specifies the list of acceptable principals that must be included in the SSH certificate for authentication to succeed.

Testing the PAM Configuration

To test the PAM configuration, you can use the pam-test tool from the pam-test repository. Below is an example of a successful test:

 1vagrant@node-1:~$ ./pam-test/pam_test auth vagrant
 2#%PAM-1.0
 3auth [success=1 default=ignore] /etc/ssh/pam_ussh.so ca_file=/etc/ssh/ca-users.pub
 4  auth requisite                  pam_deny.so
 5  auth required                   pam_permit.so
 6
 7pam_start returned [0]: Success
 8pam_authenticate returned [0]: Success
 9pam_acct_mgmt returned [0]: Success
10pam_end returned [0]: Success
11
12Authentication successful.
13vagrant@node-1:~$

This output indicates that the authentication process was successful, confirming that pam_ussh is working as expected.

Creating a CA Certificate and Issuing SSH Certificates

To create a CA certificate, use the following command:

1ssh-keygen -f ssh_host_rsa_key -N '' -b 4096 -t rsa

To issue an SSH certificate for a public key, use:

1ssh-keygen -s ca -I "$(whoami)@$(hostname --fqdn)" -n "$(whoami),superusers" -V -5m:+3650d ~/.ssh/id_rsa.pub

Make sure to include the username as a principal along with the list of authorized principals. pam_ussh requires that the certificate includes the username and, optionally, one or more of the principals listed in the authorized_principals parameter.

Conclusion

This is the culmination of MANY hours work, I hope it helps someone else facing similar challenges

P.S I'm investigating building a PAM module in Python, I know it sounds a bit gross, but the ssh / security library Go module seems to have limited support for modern certificates(Specifically SHA2-512), so when i was using a well behaved CA I was banging my head against a wall trying to figure out why pam-ussh wouldnt authorize the user, the issue was that the go library couldnt decode the SSH certificate, when i reverted back to using the CLI tool ssh-keygen it worked instantly, because the CLI tool seems to use ssh-rsa-cert-v01@openssh.com where my CA was issuing rsa-sha2-512-cert-v01@openssh.com type certificates.