- 6 min read

Allowing credential-free access to cloud resources from self-hosted Kubernetes using Pod Workload Identity

Over the last few years Kubernetes has become the go-to solution for container orchestration and application deployment. Managed solutions like Azure Kubernetes Service (AKS) or AWS Elastic Kubernetes Service (EKS) make it easy to setup production-ready Kubernetes clusters quickly without all the hassle.

One feature that is becoming more and more popular is Pod Workload Identity (Azure calls it Microsoft Entra Workload ID, AWS calls it Pod Identities). The idea is simple: Kubernetes Pods can be assigned roles in the cloud provider IAM (Microsoft Entra ID, AWS IAM) based on their identity in Kubernetes, without having to generate and distribute specific credentials (e.g. for a service principal). Workloads can use this to access services like managed PostgreSQL databases or Kafka brokers. This makes the entire setup much more secure, as there are no credentials that could be stolen or that need to be regularly rotated.

Azure and AWS both support this workload identity feature for their own managed Kubernetes solutions (AKS, EKS). But in smart factory use cases we run Kubernetes clusters also as self-managed clusters in the factory. Our go-to solution is the lightweight K3s, an open-source Kubernetes distribution that can be setup very quickly and runs single- or multi-node Kubernetes clusters on machines as small as a Raspberry Pi.

Even though workloads in these clusters will normally access other services that also run on-premise, some workloads will connect to the cloud. For example to send metrics data collected from machines onto a central Kafka cluster in the cloud or to access a central secrets manager (AWS Secrets Manager, Azure Key Vault). In this article I want to show that Pod Workload Identity is not limited to Managed Kubernetes solutions but can also be implemented relatively easily with self-hosted Kubernetes clusters, using K3s as the easy-to-use example.

How does it work

Pod Workload Identity is built on two features:

  • AWS and Azure allow configuring trust relationships with external OpenID Connect Providers to give users of these external providers permissions within the cloud provider ecosystem.
  • Pods running in Kubernetes are always associated with a Kubernetes Service Account and are provided an access token by the Kubernetes API server. The API server in turn can act as an OpenID Connect Identity Provider with all access tokens for pods being valid OIDC tokens.

With these two features a workload running in Kubernetes can use the access token granted to it by Kubernetes to authenticate against the AWS and Azure APIs and retrieve short-lived credentials.

Getting started

If you want to follow along, you will need the following:

  • A Linux or MacOS machine (or WSL2 on Windows) with Docker and K3d installed
  • kubectl installed
  • A terminal SSH client
  • Either an Azure Subscription (a free test account should work) with a Resource group and Owner permissions on that group
  • Or an AWS account with Administrator permissions (a free starter account should suffice)

Preparations on the K3s side

For K3s to act as an OpenID Connect Identity Provider its Kubernetes API server must be reachable by the cloud provider so it can access the OpenID endpoints. This normally means the API server must be exposed to the internet. If your network or setup does not allow that, you can use solutions like Cloudflare Tunnel to securely provide access to the API server. For this article I will use Pinggy as they have a free tier that does not require any installation or registration, but other services will work as well.

To expose the Kubernetes API server, first start a new Pinggy tunnel: ssh -p 443 -L4300:localhost:4300 -o StrictHostKeyChecking=no -o ServerAliveInterval=30 -t -R0:localhost:6443 a.pinggy.io \"x:localServerTls:localhost\". Take note of the generated hostname (e.g. rnhly-88-217-33-139.a.free.pinggy.link, note that it changes with each SSH reconnect, so if you loose your connection you have to start over again).

Then we can configure K3s to expose the OpenID Connect endpoints. Create a new file config.yaml and add the following content:

kube-apiserver-arg:
  - anonymous-auth=true
  - service-account-issuer=https://<hostname> # Fill in your pinggy hostname
  - service-account-jwks-uri=https://<hostname>/openid/v1/jwks  # Fill in your pinggy hostname

Now we can create a K3d cluster with that configuration: k3d cluster create witest --api-port 6443 -v $(pwd)/config.yaml:/etc/rancher/k3s/config.yaml@server:*.

Additionally, you must allow anonymous users (specifically the cloud provider IAM) to access the JWKS (JSON Web Key Sets) URL without credentials. Using this URL the provider can download the public keys to later verify tokens. To do so run the following command once the k3d cluster is created:

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: service-account-issuer-discovery-unauthenticated
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:service-account-issuer-discovery
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: Group
  name: system:unauthenticated
EOF

Without the anonymous-auth=true in the K3s config this binding would have no effect.

That is all the configuration that needs to be done on the K3s side.

Configuring Azure

For simplicity's sake we are doing all configuration via the Azure Portal, for a real setup you should properly automate this using Terraform or other infrastructure-as-code tools. First create a "User-Assigned managed identity" inside your resource group. Once created, take note of its Client ID and Tenant ID (both can be viewed under "Settings" -> "Properties").

Then you need to add a "Federated Credential" to that Managed Identity. Use the "Kubernetes accessing Azure resources" scenario. As "Cluster Issuer URL" use the Pinggy hostname including https://, for "Namespace" use default and for "Service Account" azure-cli. The "Audience" you can leave with its default value.

As a last step we need to give the Managed Identity some permissions. For example assign it the "Contributor" role on your resource group: Open your resource group, go to "Access Control (IAM)", then "Add role assignment". Switch to "Privileged administrator roles" and select "Contributor". Under "Members" switch to "Managed Identity" and use "Select Members" to add your Managed Identity you created for K3s.

Now we are ready to use the Managed Identity from Kubernetes.

Accessing Azure from a Kubernetes Pod

To actually use the Managed Identity from a Pod we first need to create the Kubernetes service account: kubectl create sa azure-cli.

Then you can create a pod to use the service account and run the Azure CLI (insert the Client ID and Tenant ID you noted down earlier):

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: azure-cli
spec:
  containers:
  - name: cli
    image: mcr.microsoft.com/azure-cli
    command:
      - /bin/bash
      - -c
      - tail -f /dev/null
    env:
      - name: AZURE_CLIENT_ID
        value: <client-id> # Insert the Client ID of the Managed Identity
      - name: AZURE_TENANT_ID
        value: <tenant-id> # Insert the ID of your Entra ID Azure Tenant
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: oidc
  serviceAccountName: azure-cli
  volumes:
  - name: oidc
    projected:
      sources:
      - serviceAccountToken:
          path: oidc
          expirationSeconds: 7200
          audience: api://AzureADTokenExchange
EOF

Once the pod is running, connect to it: kubectl exec -it azure-cli -- bash. You can use the Kubernetes service account of the pod to log into the Azure CLI: Run az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token $(cat /var/run/secrets/tokens/oidc). The response should be a JSON document showing information about the Azure subscription you logged into. To test your permissions, run a command like az group list, which will show the resource group you have authorized the Managed Identity for.

If this works, any other tool or custom written application that uses the Azure SDK for authentication will work as well.

Make sure to remove the Managed Identity and its role assignment from Azure afterwards.

Configuring AWS

The same functionality not only works with Azure but also with AWS. There we need to create two resources:

First create an identity provider: In the AWS Console navigate to the IAM service and go to Identity Providers. Create a new one. As type, select OpenID Connect, as Provider URL and Audience use the hostname for the API server (e.g. https://rnhly-88-217-33-139.a.free.pinggy.link, including the https:// part). Then select Add provider.

Then create a new role with a custom trust policy. Use the following as trust policy document:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<aws-account-id>:oidc-provider/<apiserver-hostname>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "<apiserver-hostname>:sub": "system:serviceaccount:default:aws-cli"
                }
            }
        }
    ]
}

Replace <apiserver-hostname> with the hostname of the API server but without https (e.g. rnhly-88-217-33-139.a.free.pinggy.link). Give the role some permissions, e.g. AmazonS3FullAccess.

With that AWS is configured and we can use the role.

Using AWS from a Kubernetes Pod

To actually use the workload identity we first need to create the Kubernetes service account: kubectl create sa aws-cli.

Then you can create a pod to use the service account and run the AWS CLI:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: aws-cli
spec:
  containers:
  - name: cli
    image: amazon/aws-cli:2.17.4
    command:
      - /bin/bash
      - -c
      - tail -f /dev/null
    volumeMounts:
    - mountPath: /var/run/secrets/tokens
      name: oidc
    env:
    - name: AWS_WEB_IDENTITY_TOKEN_FILE
      value: /var/run/secrets/tokens/oidc
    - name: AWS_ROLE_ARN
      value: arn:aws:iam::<aws-account-id>:role/<name-of-the-role>  # fill in your aws account id and the name of the role you created
  serviceAccountName: aws-cli
  volumes:
  - name: oidc
    projected:
      sources:
      - serviceAccountToken:
          path: oidc
          expirationSeconds: 7200
          audience: https://<apiserver-hostname> # fill in your API server hostname
EOF

Once the pod is running, connect to it: kubectl exec -it aws-cli -- bash. In contrast to the Azure CLI, the AWS CLI will automatically obtain credentials based on the configured environment variables. To verify the identity federation works, run aws sts get-caller-identity. Then, to test permissions you can run something like aws s3api list-buckets.

If this works, any other tool or custom written application that uses the AWS SDK for authentication will work as well.

Make sure to delete the role and identity provider in AWS after you finish.

Conclusions

Hopefully this article has given you the insight that setting up workload identity for self-hosted Kubernetes clusters is not hard. And if you are using managed Kubernetes solutions from cloud providers, you now at least know what is going on behind-the-scenes.

Workload Identity for Kubernetes is a feature that has a high potential to simplify usage and increase security at the same time. Once it is configured on the Kubernetes and cloud provider sides, assigning permissions and using them from within a pod is easy. Users don't have to distribute or rotate any credentials, and admins can sleep easier knowing there are no credentials that could get stolen or misused.