Kubernetes: SSO for web applications with Keycloak, Kerberos and Ingress Controller (and Pitfalls and Traps)

tl;dr

This article describes the basic setup of web applications in Kubernetes with Keycloak for user authorization via Single-SignOn with Kerberos. The focus is on glueing the components together, so basic understanding of running applications in Kubernetes is a prerequisite.
Web applications run in a Kubernetes cluster and redirect the web browser to the Keycloak instance for authorization. The Keycloak instance also runs in the Kubernetes cluster and will only be accessed through an Ingress-entity via host path served by an Ingress controller which is accessible from the outside world. The Keycloak server uses a Kerberos Principal with keytab for accessing the Kerberos server (outside the cluster) for authorization of a user. The DNS setup for this scenario is key: Web browsers using Kerberos tickets, triggered by an HTTP header Authorization: Negotiate , always look up the A-record of the authorization server and use this hostname in their answer in their Authorization section. Therefore, whatever the DNS CNAME of the Keycloak server is, the Kerberos principal used by Keycloak must match the A-record of the Kubernetes Ingress-controller of the cluster, and the keytab must be created accordingly.

Introduction

Keycloak as an authentication and authorization server is a widely deployed solution for securing applications and websites. In our environment we use it for securing Springboot applications deployed in a bare-metal Kubernetes cluster. This seems to be a straightforward installation as there are many tutorials available which describe the setup of Keycloak for this purpose. However, although the setup of each single component is well-known, getting them to play as a team in K8s can be challenging and we had a really hard time to make this work.

This article is intended to serve all admins, dev-ops people and developers as a guide for glueing together a scenario like this. Although there are a ton of tutorials of how to connect web services with Kerberos and have browser use Kerberos tickets, the special setup we’re dealing with lays out some traps which potentially cost you a lot of debugging time which we already invested in this setup. Therefore we’d like to share out knowledge.

I assume that you are familiar with securing applications with Keycloak, and know your way around K8s for deploying and running applications. Additionally I assume that you have a working Kerberos setup available. I will not go into details of these topics.

Prerequisites

The following prerequisites were given:

• Users work under Windows 10 and with the Chrome browser. Authentication happens via Active Directory (AD), and the browser is prepared to use Kerberos tokens for authentication. This already works well with numerous web-pages under Apache outside of our K8s cluster.
• Our web-applications that are developed in-house are being deployed in an on-premise K8s cluster. We use Springboot as backend and Angular as frontend (although this doesn’t matter).
• User authentication and authorization is carried out with Keycloak, our applications are prepared accordingly. This means that for authentication the application redirects the user to Keycloak. Keycloak then uses Kerberos in order to provide SSO, and redirects the user to the original website. Ideally, the user doesn’t notice the login process at all.
• The Keycloak server also runs in the K8s cluster as a deployment.
• Access to all web-applications and also to the Keycloak-server is provided via an NGINX-Ingress-controller in K8s. The ingress is the only entry into the cluster for end users, and it has a dedicated (external) IP address with a matching A record in the DNS.
• Single applications can be accessed via a CNAME record in the DNS, which then resolves to the IP of the ingress controller. However, by the host name of the original request the ingress controller can decide to which service it has to forward the request to.
  Example:
  The ingress controller has the IP 10.1.20.50/24 to the outside world, and an A record of k8s-ingress.company.com. A web service service.company.com would be a CNAME to k8s-ingress.company.com, and thus resolve to the same IP. However, the original request contains service.company.com as a host name in the header and can therefore serve as directive for the ingress controller how to forward the request.
  I’m focusing on this part because it’s vitally important for understanding the mechanisms below.

General flow

Here’s a short outline of the authentication flow in order to understand the general cooperation of all the components:

• With logging in under Windows 10 the user retrieves a Kerberos ticket.
• The users accesses the application web site which sends back a redirect per HTTP code 302, the redirect points to the Keycloak server for authentication.
• First access of the Keycloak redirect-URL usually results in a 401 HTTP error, saying that the user is not authenticated. But, at the same time, the server offers to negotiate means for authention with the browser by sending the HTTP header Authorization: Negotiate.
• This offer is picked up by the browser and it generates a Kerberos token and includes it in the Authorization header of the next request. Important: Besides information about the current user this token also includes the A record of the FQDN of the web server that is being requested. Usually the FQDN of the web service is a CNAME for the A record of the ingress controller, so in fact the FQDN included in the token is the one of the ingress controller. This fact is important for the configuration of the setup.
• The Keycloak server extracts the aforementioned FQDN and uses it as the Kerberos principal for communication with the Kerberos server. Of course, for this to work, there must be a principal configured in the KDC which matches this FQDN.
• Once the Keycloak server gets the “green light” for the user to access the application, it redirects the browser to the original application.
  If authorization wasn’t successful the Keycloak server will offer other means of authentication (e.g. login dialog) if configured. If not, the authentication will fail.

Example tour

For describing the details it’s good to see it in an example. The following diagram outlines the setup in the K8s cluster:

Setting

• Keycloak: keycloak.company.com
• K8s Ingress: k8s-ingress.company.com
• Protected website: website.company.com
• Kerberos realm: COMPANY.COM

DNS

• k8s-ingress.company.com
  A record for 10.1.20.50/24
• keycloak.company.com
  CNAME for k8s-ingress.company.com
• website.company.com
  CNAME for k8s-ingress.company.com

Browsers

Browsers need to be prepared in order to use Kerberos tickets for authentication. There are enough tutorials for making browsers and AD/Kerberos together, I’ll only give an example for Chrome, other browser have the same mechanics: The latest version of Chrome uses policies for configuring, i.e. check chrome://policy in order to check your current setting. What needs to be configured is

• AuthSchemes
  This tells the browser which schemes to use when confronted with a header Authorization: Negotiate. At least negotiate must be configured here.
• AuthServerAllowlist
  This entry contains the servers with which the browser should perform Authentication Negotiation. The Keycloak server keycloak.company.com must be entered here (or alternatively a wildcard expression for the whole domain, e.g. *.company.com).

Microsoft AD/Kerberos

To have Keycloak talk to the AD/Kerberos server we need a Kerberos principal and a keytab. In this case the principal must be k8s-ingress.company.com, i.e. HTPP/k8s-ingress.company.com@COMPANY.COM . Create a keytab file for this principal, e.g. http_k8s-ingress.keytab and configure Keycloak accordingly.

Keycloak

Keycloak needs a native KRB5 installation in order to work with Kerberos (details can be found in the current Keycloak docs). So, for Keycloak running in K8s you might need to prepare your deployment, especially when using pre-built Docker-images you might have to amend those in order to install the necessary tools and libraries. The official Keycloak image from RedHat uses microdnf as package manager, and with it all neccessary Kerberos stuff can be installed with microdnf install krb5-workstation. Make sure that the realm COMPANY.COM is configured according to your KRB5-setup in /etc/krb5.conf.

Create a realm in Keycloak for your application and make sure that they can talk to each other. Without Kerberos integration in the default setup of Keycloak you should see a login dialog popping up when accessing your application. The details for connecting web applications to Keycloak is beyond the scope of this article, but if you came here because the title is interesting you already know what to do.

Next, create a user federation in Keycloak. Use either kerberos or ldap. If you choose ldap you also need to configure “Use Kerberos” further down the page. In the Kerberos section define Kerberos-Realm COMPANY.COM and principal HTPP/k8s-ingress.company.comto use when contacting the Kerberos server. Additionally use the keytab file http_k8s-ingress.keytab mentioned before and tell Keycloak where to find it.
Note: The principal is key here. Keycloak will extract the principal from the token offered by the browser, and that is the DNS A record of the ingress controller. Therefore the principal must match the DNS A record of the ingress controller of your cluster, otherwise Keycloak will never try to authenticate via Kerberos.

There are a few challenges when Keycloak is installed in a K8s cluster. The Kerberos config in /etc/krb5.confneeds to be injected in the pod, as well as the keytab file http_k8s-ingress.keytab. We solved this by using configmaps for both files and subpath-mounts for them in the container section of the Keycloak-deployment.
Excerpt from the deployment:

[...]
volumeMounts:
 - name: keytabvolumne
   mountPath: /etc/keycloak.keytab 
   subPath: data.keytab
   readOnly: true      
 - name: krb5volumne
    mountPath: /etc/krb5.conf 
    subPath: krb5.conf
    readOnly: true
[...]
 volumes:    
 - name: keytabvolumne
   configMap:
     name: keytab-config 
 - name: krb5volumne
   configMap:
     name: krb5-config

and from the config map, for example krb5-config :

[...]
data:
  krb5.conf: |-
    [libdefaults]
      default_realm = COMPANY.COM
    [realms]
      WL.ADS = {
            kdc = kdc.company.com
            admin_server =kdc.company.com
      }

Experience

While setting this up we were able to find information about the single topics covered in this articel, e.g. installing Keycloak, connecting web applications to it, and use Kerberos as SSO server. However, the key information for this special scenario used in a K8s cluster was nowhere to be found. I hope you found this article useful and if you like you can leave a comment.