- 4 min read

Letting Github and Gitlab Pipelines access Cloud Provider APIs without credentials

In an earlier post I have already discussed how we can grant Kubernetes Pods access to cloud provider APIs without credentials. Today I want to discuss how we can do the same with popular CI/CD tools Github Actions and Gitlab CI.

Pipelines are an important part of my daily doing at MaibornWolff. They test and build my code and handle my deployments. Most customer projects nowadays are built at least in part in the cloud. This means pipelines need to interact with cloud provider APIs. Be that to log into a container registry to push a freshly built image, or to perform infrastructure-as-code tasks with tools like Terraform.

For the longest time this meant creating a service user (IAM User in AWS, Service Principal in Azure) and managing and distributing credentials for it to the pipelines (as CI/CD Variables in Gitlab or Actions Secrets in Github).

But this approach is not ideal, both from a security and usability perspective. Others can extract and (mis)use the credentials and regularly rotating them just creates manual effort. A way better way to handle this scenario is to create a trust relationship between the Pipeline system and the cloud provider. This allows running pipeline jobs to assume roles in the cloud API without having to deal with credentials.

Just like with Kubernetes Workload Identity the trick is accomplished using OpenID Connect (OIDC). Both Github and Gitlab act as OIDC Providers for their pipeline jobs, issuing identities and job tokens for each running job. In AWS and Azure we can configure a federation with the pipeline system and use that trust relationship to grant the pipeline job permissions for the cloud provider API.

In the following paragraphs I want to show how this can be done for Github and Gitlab using both AWS and Azure as cloud providers. I will show the AWS and Azure specific configuration using Terraform. If you want to follow along, you will need:

  • Terraform installed and basic working knowledge of using Terraform with AWS/Azure
  • AWS and Azure CLIs installed
  • AWS account and user with Admin permissions for it
  • Azure subscription and user with Owner permissions for it
  • Active sessions for these users in the AWS and Azure CLIs
  • Accounts on Github/Gitlab so that you can create repos (the free plan is enough)

Configuring AWS

Let's start with AWS. This section is based on the official Github documentation. The first step is to configure Github as an OIDC provider:

# Configure the AWS provider
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}
provider "aws" {
  region = "eu-central-1"
}

# Create an OIDC provider
resource "aws_iam_openid_connect_provider" "github" {
  url = "https://token.actions.githubusercontent.com"

  client_id_list = ["sts.amazonaws.com"]

  # https://github.blog/changelog/2023-06-27-github-actions-update-on-oidc-integration-with-aws/
  thumbprint_list = [
    "6938fd4d98bab03faadb97b34396831e3780aea1",
    "1c58a3a8518e8759bf075b76b750d4f2df264fcd"
  ]
}

Then we can create an IAM role to be assumed by the Github runner and give it some permissions (I use access to S3 as an example):

# Create a role
resource "aws_iam_role" "github" {
  name = "github-actions"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Principal = {
          # Only federated tokens from Github are allowed to assume this role
          Federated = "arn:aws:iam::<account-id>:oidc-provider/token.actions.githubusercontent.com"
        }
        Condition = {
          StringLike = {
            # Only runners for this repository and the main branch are allowed to use the role
            # Replace the tail with a wildcard `ref:*` to allow jobs running on all branches
            "token.actions.githubusercontent.com:sub": "repo:swoehrl-mw/iac-aws:ref:refs/heads/main"
          }
          StringEquals = {
            # The default audience as expected by AWS 
            "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
          }
        }
      },
    ]
  })
}

# Assign permissions to the role via a policy
resource "aws_iam_role_policy" "github_s3" {
  role = aws_iam_role.github.id

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        # Full access to S3
        Action = ["s3:*"]
        Effect   = "Allow"
        Resource = "*"
      },
    ]
  })
}

# The role arn needs to be provided in the Pipeline
output "role_arn" {
  value = aws_iam_role.github.arn
}

After a terraform init and terraform apply the configuration on the AWS side is complete.

Accessing AWS from Github

Now we can create a repository in Github (in terraform I used swoehrl-mw/iac-aws, adapt it to match yours) and add a workflow to it (e.g. as .github/workflows/dummy.yaml):

on: push

permissions:
  contents: read   # For git clone access
  id-token: write  # To get the job token

jobs:
  dummy:
    runs-on: ubuntu-latest
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          # Fill in the ARN from the terraform output
          role-to-assume: arn:aws:iam::<account-id>:role/github-actions
          aws-region: eu-central-1
      - run: |
          # Because we gave the role S3 permissions this call will succeed
          aws s3api list-buckets

Once you commit and push the workflow definition, Github will start a job and you can verify that it succeeds and prints a list of S3 buckets in your account.

After the aws-actions/configure-aws-credentials call, credentials are configured and can be used with the AWS CLI, with Terraform or any other tool that uses the AWS SDK or knows how to retrieve credentials from the AWS CLI.

Configuring Azure

For Azure the situation is a bit different but conceptually similar. Here instead of an IAM Role with an AssumeRole policy a Managed Identity with a configured Federated Credential is used. Again this section is based on the official Github documentation.

# Configure the Azure provider
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.10.0"
    }
  }
}
provider "azurerm" {
  subscription_id = "your-subscription-id"
  features {}
}
data "azurerm_subscription" "current" {
}

# Create a new resource group
resource "azurerm_resource_group" "citrust" {
  name     = "citrust"
  location = "West Europe"
}
# Create a managed identity for Github/Gitlab
resource "azurerm_user_assigned_identity" "cicd" {
  location            = azurerm_resource_group.citrust.location
  name                = "github"
  resource_group_name = azurerm_resource_group.citrust.name
}

# Add a Federated credential for the managed identity
resource "azurerm_federated_identity_credential" "github" {
  name                = "github"
  resource_group_name = azurerm_resource_group.citrust.name
  # Standard audience as requested by Azure
  audience            = ["api://AzureADTokenExchange"]
  issuer              = "https://token.actions.githubusercontent.com"
  parent_id           = azurerm_user_assigned_identity.cicd.id
  # Only runners from this repo using the main branch are granted access
  subject             = "repo:swoehrl-mw/iac-azure:ref:refs/heads/main"
}

# Assign permissions to the managed identity
resource "azurerm_role_assignment" "cicd" {
  scope                = data.azurerm_subscription.current.id
  role_definition_name = "Contributor"
  principal_id         = azurerm_user_assigned_identity.cicd.principal_id
}

# Values that must be provided in the pipeline
output "client_id" {
  value = azurerm_user_assigned_identity.cicd.client_id
}
output "tenant_id" {
  value = data.azurerm_subscription.current.tenant_id
}

Again after a terraform init and terraform apply the configuration for the cloud provider is done.

Accessing Azure from Github

Now we can again create a repository in Github (in Terraform I have used swoehrl-mw/iac-azure, adapt to match yours) and add a workflow to it:

on: push

permissions:
  contents: read   # For git clone access
  id-token: write  # To get the job token

jobs:
  dummy:
    runs-on: ubuntu-latest
    steps:
      - uses: azure/login@v1
        with:
          # Replace with the client_id from the terraform output
          client-id: client-id
          # Replace with the ID of your tenant from the terraform output
          tenant-id: tenant-id
          # Replace with your Azure subscription ID
          subscription-id: your-subscription-id
      - run: |
          az account show
          az group list

Once you commit and push the workflow definition, Github will start a job and you can verify that it succeeds and prints account information and a list of your resource groups.

In contrast to AWS the Azure Federated Credential configuration is limited in that it does not support wildcards. This makes setups with pipelines running from different branches (e.g. feature branches) practically impossible as you would have to add a Federated Credential for each feature branch name. And you are limited by the fact that a Managed Identity can only have up to 20 Federated Credentials. So for Azure this setup is only usable if you deploy from a limited and pre-defined list of branches.

Doing it all with Gitlab

Gitlab works similar to Github. There is official Gitlab documentation for AWS and Azure.

Note: If you are using a self-hosted Gitlab instance, replace any mention of gitlab.com with the host of your instance (e.g. gitlab.mycompany.com).

From the Terraform code I only show the parts that are added/changed in contrast to the setup for Github.

For AWS we need to create a new OIDC Provider and role that uses it:

resource "aws_iam_openid_connect_provider" "gitlab" {
  url = "https://gitlab.com"

  client_id_list = ["sts.amazonaws.com"]

  thumbprint_list = [
    # For simplicity we skip the thumbprint with this dummy value
    "ffffffffffffffffffffffffffffffffffffffff",
  ]
}

resource "aws_iam_role" "gitlab" {
  name = "gitlab-ci"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRoleWithWebIdentity"
        Effect = "Allow"
        Principal = {
          Federated = "arn:aws:iam::<account-id>:oidc-provider/gitlab.com"
        }
        Condition = {
          StringLike = {
            # You can also use a wildcard here, e.g. project_path:swoehrl-mw/iac-aws:*
            "gitlab.com:sub": "project_path:swoehrl-mw/iac-aws:ref_type:branch:ref:main"
          }
          StringEquals = {
             "gitlab.com:aud": "sts.amazonaws.com"
          }
        }
      },
    ]
  })
}

The role policy with permissions for S3 stays the same.

In the Gitlab repo (I am again using swoehrl-mw/iac-aws as example) we need a .gitlab-ci.yaml like the following:

variables:
  # Add the ARN for the role from the terraform output
  AWS_ROLE_ARN: arn:aws:iam::<account-id>:role/gitlab-ci

dummy:
  stage: dummy
  image: 
    name: amazon/aws-cli:latest
    entrypoint: [""] # Override to get a normal shell
  id_tokens:
    # This tells Gitlab to provide a token with the given audience (must match what is configured at the cloud provider)
    GITLAB_OIDC_TOKEN:
      aud: sts.amazonaws.com
  script:
    # This command is needed as the AWS CLI does not automatically request credentials using assume-role
    - >
      export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s"
      $(aws sts assume-role-with-web-identity
      --role-arn ${AWS_ROLE_ARN}
      --role-session-name "Gitlab"
      --web-identity-token ${GITLAB_OIDC_TOKEN}
      --duration-seconds 3600
      --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]'
      --output text))
    - aws sts get-caller-identity
    - aws s3api list-buckets

For Azure we just need to create a different Federated Credential as follows:

resource "azurerm_federated_identity_credential" "gitlab" {
  name                = "gitlab"
  resource_group_name = azurerm_resource_group.citrust.name
  audience            = ["api://AzureADTokenExchange"]
  issuer              = "https://gitlab.com"
  parent_id           = azurerm_user_assigned_identity.cicd.id
  subject             = "project_path:some-group/iac-azure:ref_type:branch:ref:main"
}

Then in our Gitlab CI pipeline we need to have a job like the following:

variables:
  AZURE_CLIENT_ID: "your-client-id"
  AZURE_TENANT_ID: "your-tenant-id"

dummy:
  stage: dummy
  image: mcr.microsoft.com/azure-cli:latest
  id_tokens:
    GITLAB_OIDC_TOKEN:
      aud: api://AzureADTokenExchange
  script:
    - az login --service-principal -u $AZURE_CLIENT_ID -t $AZURE_TENANT_ID --federated-token $GITLAB_OIDC_TOKEN
    - az account show
    - az group list

Conclusion

It is unfortunate that Azure has still not implemented wildcard support for defining the subject of a Federated Credential as that severely limits the possibilities of using CI Trust with Github and Gitlab. AWS is clearly the more flexible solution here.

Setting up a trust relationship between a cloud provider (AWS or Azure) and a pipeline platform (Github or Gitlab) is actually very easy. It requires the initial creation of a few resources on the cloud side, but this only needs to be done once and can easily be integrated into an initial account or platform bootstrap process. With the OIDC Federation set up there is no need for any distribution or rotation of credentials. A big improvement for both security and ease-of-use.