- 4 min read

Securing self-hosted Kubernetes clusters with SSO using OIDC and K3s

Over the last few years Kubernetes has become the go-to solution for container orchestration and application deployment. Managed solutions like Azure Kubernetes Service (AKS) or AWS Elastic Kubernetes Service (EKS) make it easy to setup production-ready Kubernetes clusters quickly without all the hassle.

One feature widely used by Enterprises is the SSO integration. Instead of manually generating tokens or client certificates for users to authenticate against the Kubernetes API, AKS and EKS both integrate into the IAM solutions of the respective providers (Microsoft Entra ID and AWS IAM). And use them to provide central authentication and authorization. This way no separate user accounts need to be set up for Kubernetes, and permissions can be granted based on roles or group memberships, which in most companies are already used. It also makes offboarding simple, without an active SSO account a user will simply no longer be able to access the Kubernetes API, no need to deactivate any Kubernetes logins separately.

In Smart factory use cases running Kubernetes in the cloud using AKS or EKS is not enough. We need Kubernetes close to the machines, in the factory, and ideally lightweight. Our go-to solution for that is K3s, an open-source Kubernetes distribution that can be setup very quickly and runs single- or multi-node Kubernetes clusters on machines as small as a Raspberry Pi. It seems with using K3s we lose the integrated SSO support that the cloud-provider solutions provide. But that is not the case.

In this article I want to show how easy it is to connect K3s with an OpenID Connect-compatible Identity Provider to provide SSO login capabilities for the Kubernetes API.

Getting started

If you want to follow along, you will need the following:

  • A Linux or MacOS machine (or WSL on Windows) with Docker and K3d installed (or a running K3s cluster)
  • Kubectl installed
  • The kubectl kubelogin plugin installed
  • A gitlab.com account (can be created for free) or another system that provides an OIDC Identity Provider (e.g. Entra ID, Google Identity Platform), see the kubelogin setup guide for options

The setup has four parts:

  1. Configuring the OIDC provider and an OIDC client/application
  2. Configuring K3s to verify OIDC tokens against an issuer
  3. Granting permissions for OIDC users in Kubernetes
  4. Configuring kubectl to use OIDC login

Configuring the OIDC provider

The first step is to configure on OAuth/OIDC client/application with your OIDC provider. I'm using Gitlab as accounts can be created for free and the setup is very simple for testing (Note: Github does provide OAuth applications but is not OIDC compatible):

  1. Open the Applications page in your user settings
  2. Add a new application
  3. Give it a name like "K3s SSO Login" and add http://localhost:8000 and http://localhost:18000 to the Redirect URIs
  4. Disable the Confidential setting
  5. For the scopes, select openid, profile and email
  6. Save the application, note down the Application ID (which is the Client ID), the secret you can ignore

If your OIDC provider does not support public applications (that do not need a client secret) you can also create a confidential one, but you will have to share the client secret with all users.

Configuring K3s for OIDC

Once the OIDC provider is set up, we can configure K3s. Create a new file config.yaml and add the following content:

kube-apiserver-arg:
  - oidc-issuer-url=https://gitlab.com
  - oidc-client-id=<client-id> # fill in the application id from github
  - oidc-username-claim=preferred_username
  - oidc-groups-claim=groups
  - "oidc-username-prefix=oidc:"
  - "oidc-groups-prefix=oidc:"

Then we can create a quick K3d cluster for testing: k3d cluster create oidctest -v $(pwd)/config.yaml:/etc/rancher/k3s/config.yaml@server:*.

If you are using an existing K3s cluster, you need to add the content to the file /etc/rancher/k3s/config.yaml (or create it if it does not exist) and restart K3s (systemctl restart k3s).

Granting Kubernetes permissions

Now that K3s is configured, we need to use RBAC to grant our OIDC user permissions. Make sure your kubectl is pointing to the newly created cluster, then run:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: oidc-admin
subjects:
  - kind: User
    name: oidc:<your-gitlab-username> # fill in your gitlab username here
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io
EOF

This will grant your OIDC user cluster-admin permissions in the cluster. If your Identity Provider also adds groups to the OIDC token, you can also authorize an entire group by using a subject of kind: Group and name: oidc:<group-name>.

Configuring kubectl

As the last step we need to tell kubectl to retrieve an OIDC token and authenticate against the Kubernetes API with it. To do that we need to modify our kubeconfig. Retrieve it by running k3d kubeconfig get oidctest > kubeconfig-oidc. Edit the file and switch out the configured user with the following:

users:
- name: admin@k3d-oidctest # If your kubeconfig uses a different user name, use that here
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      command: kubectl
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://gitlab.com
      - --oidc-client-id=<client-id> # fill in the application id from github
      - --oidc-extra-scope=openid,profile,email # You might need to adapt these depending on your Identity Provider
      #- --oidc-client-secret=<client-secret>

If your provider only supports confidential clients, comment in the last line and fill in the client secret of your OIDC client/application.

Putting it all together

Now we can test it out. Point your kubectl to our modified kubeconfig by running export KUBECONFIG=$(pwd)/kubeconfig-oidc. Then run any kubectl command, e.g. kubectl get pods. An authorization page of Gitlab (or your Identity Provider) will open in your browser. Login and approve the application. Once you see an Authenticated page, you can return to your terminal. If everything went well, you will see a normal kubectl output.

If you get an Unauthenticated error, check the Kubernetes logs (for K3d docker logs k3d-oidctest-server-0) for problems. Often the client-id, issuer-url or scopes don't match. If instead you get a Forbidden error, then most likely the username in the ClusterRoleBinding does not match what the Identity Provider reports in the OIDC token. You can find the token in your ~/.kube/cache/oidc-login. Open the file with a long random name in there and copy out the value of the id_token field in the JSON. You can decode it with a tool like jwt.io. Verify the value of preferred_username matches the ClusterRoleBinding. If your provider sends the username in another field, you will need to change the oidc-username-claim or oidc-groups-claim parameters in the K3s config.yaml to match and restart.

Conclusions

Even without the benefits of a Managed Kubernetes like AKS or EKS, configuring Kubernetes with SSO is no magic feat and in fact quite easy. The configuration options presented in this article are not k3s-specific but are in fact part of vanilla Kubernetes so can be used with basically every Kubernetes distribution, as long as the Kubernetes API server can be configured. As OpenID Connect is an open standard and used by many identity providers, it gives a wide variety of choices (and if your provider does not support it, maybe a connector like Dex can help). Putting both together makes an easy and powerful mechanism to configure and use SSO login with self-managed Kubernetes.