GCP Workload Identity Federation with AWS ECS Tasks

Introduction

GCP Workload Identity Federation (WIF) allows external workloads, such as those running on AWS, Azure, GitHub, etc., to interact with GCP resources without the need for long-lived credentials (service account keys). This allows developers to connect to GCP through any identity provider (IdP) that supports OpenID Connect (OIDC) or SAML 2.0, eliminating the overhead of managing sensitive service account keys. This is a stellar security feature that could use a much broader adoption in the industry than the status quo.

When talking about AWS workloads, GCP auth libraries provide native WIF support only for EC2 instances. For ECS tasks, GCP auth libraries do not perform WIF out of the box. Let’s dive into why that is and how you can perform WIF from AWS ECS tasks.

TL;DR Please - How does GCP Workload Identity Federation (WIF) Work?

GCP WIF authenticates external workloads by exchanging workloads’ credentials for Google access tokens through the GCP Security Token Service (STS)¹. External workloads can then access GCP resources through two possible flows:

Using federated identities, i.e., accessing GCP resources directly on behalf of a principal identity or
By service account impersonation — where your workload assumes a service account, and that service account should have the necessary permissions to access the GCP resource

Although GCP recommends you use federated identities and grant direct access to GCP resources, there are many API limitations on what these federated identities can even access. So there are certainly use cases where service account impersonation fits the bill better.

For a detailed explanation of how GCP WIF works, check out this excellent article. This flowchart summarizes WIF flow for an OIDC-compliant IdP with access through service account impersonation.

GCP WIF Flow with Service Account Impersonation — From https://blog.salrashid.dev/articles/2021/understanding_workload_identity_federation/

Primer: WIF for AWS Workloads

For AWS workloads to connect to GCP, they need to exchange their temporary AWS credentials for Google access tokens through the GCP STS endpoint. Let’s zoom in and see how exactly that pans out.

GCP STS endpoint is available at https://sts.googleapis.com/v1/token and expects a POST request with a JSON body, containing parameters for WIF configuration (based on rfc8693). One of these parameters is subject_token, which contains the external credential information, verifiable through the IdP. What this means is that the GCP STS endpoint should be able to use the subject_token and verify that the external credential is indeed valid and issued by the trusted IdP (AWS in this case).

Sample request body for GCP STS Endpoint:

{
  "audience": "//iam.googleapis.com/projects/PROJECT_ID/locations/global/workloadIdentityPools/POOL_ID/providers/PROVIDER_ID",
  "grant_type": "urn:ietf:params:oauth:grant-type:token-exchange",
  "requested_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "scope": "scope1 scope2",
  "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
  "subject_token": <SUBJECT_TOKEN>
}

Plug in a subject_token value in the request body, and the STS endpoint returns the GCP access token.

Sample successful response from GCP STS Endpoint:

{
  "access_token": "XXXXXXXXXXX",
  "issued_token_type": "urn:ietf:params:oauth:token-type:access_token",
  "token_type": "Bearer",
  "expires_in": 3600,
  "scopes": "scope1 scope2"
}

Constructing Subject Token Value

For AWS, the subject_token value is an AWS signed request (SigV4) for GetCallerIdentity endpoint: https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15. This is essentially AWS telling GCP: “Hey, you can pass this request to me. I will validate and return the details of the AWS identity (user/role) who signed this request”.

Sample signed request to AWS GetCallerIdentity:

POST / HTTP/1.1
Host: sts.amazonaws.com
Content-Type: application/x-www-form-urlencoded
Authorization: AWS4-HMAC-SHA256 Credential=AKIAI12QH8AHCEXAMPLE/20160301/us-east-1/sts/aws4_request,
 SignedHeaders=host;user-agent;x-amz-date;x-amz-security-token,
 Signature=1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
X-Goog-Cloud-Target-Resource: //iam.googleapis.com/projects/$PROJECT_ID/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID
X-Amz-Date: 20250126T215751Z
X-Amz-Security-Token: XXXXXXXXXXX

Action=GetCallerIdentity&Version=2011-06-15

AWS workloads generate this signed request using their temporary AWS credentials. This request is serialized and passed into the subject_token value, the serialized version looking something like:

{
  "url": "https://sts.us-east-1.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15",
  "headers": [
 {
      "value": "//iam.googleapis.com/projects/$PROJECT_ID/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID",
      "key": "x-goog-cloud-target-resource"
 },
 {
      "value": "20250126T215751Z",
      "key": "x-amz-date"
 },
 {
      "value": "AWS4-HMAC-SHA256 Credential=AKIAI12QH8AHCEXAMPLE/20160301/us-east-1/sts/aws4_request,SignedHeaders=host;user-agent;x-amz-date;x-amz-security-token,Signature=1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
      "key": "Authorization"
 },
 {
      "value": "sts.us-east-1.amazonaws.com",
      "key": "host"
 },
 {
      "value": "XXXXXXXXXXX",
      "key": "x-amz-security-token"
 }
 ],
  "method": "POST",
  "body": ""
}

To summarize, GCP STS endpoint receives the request from AWS workload, verifies the AWS role by passing the GetCallerIdentity signed request (subject_token’s value) back to AWS — in return receiving information about the role ARN and AWS Account ID. Voila! At this point, GCP has authenticated the AWS workload, and it issues a GCP access token.

GCP provides client libraries (JAVA, Python, Node.js, Go, etc.) to help developers implement WIF functionality in their applications. So you should be able to leverage these SDKs in your AWS workloads, and everything must work seamlessly, correct? Welp, not quite!

WIF Configuration, AWS EC2 and ECS - The Problem

While configuring WIF in your GCP project, you perform a set of actions:

Setting up workload identity pool and provider
Defining attribute mappings and conditions
Allowing external workload to access resources through either direct federated identity access or service account impersonation

Once the configuration is complete, you fetch a credential configuration file². This configuration file is used by the google-auth client libraries to handle the whole WIF flow, that we discussed above, for you.

Sample credential configuration file:

{
  "type": "external_account",
  "audience": "//iam.googleapis.com/projects/$PROJECT_ID/locations/global/workloadIdentityPools/$POOL_ID/providers/$PROVIDER_ID",
  "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request",
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/$EMAIL:generateAccessToken",
  "token_url": "https://sts.googleapis.com/v1/token",
  "credential_source": {
    "environment_id": "aws1",
    "region_url": "http://169.254.169.254/latest/meta-data/placement/availability-zone",
    "url": "http://169.254.169.254/latest/meta-data/iam/security-credentials",
    "regional_cred_verification_url": "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06-15",
    "imdsv2_session_token_url": "http://169.254.169.254/latest/api/token"
 }
}

Let’s break down this configuration file:

Notice that this credential configuration file contains parameters that are required in the GCP STS endpoint request (audience, subject_token_type, etc.).
token_url value is the GCP STS endpoint URL.
service_account_impersonation_url parameter indicates that this WIF access is through service account impersonation.
credential_source parameter is used to generate the subject_token value.
- credential_source.regional_cred_verification_url is the AWS GetCallerIdentity API endpoint.

Eagle-eyed readers would also notice that the endpoint 169.254.169.254 in credential_source, is the EC2 Instance Metadata Service (IMDS) URL — used to fetch instance profile AWS role credentials. EC2 role credentials are fetched by AWS SDKs/CLI automatically while in use. Alternatively, you can also retrieve the role credentials manually by sending HTTP requests.

GCP SDKs (google-auth client libraries) are designed to use this credential_source configuration by default and fetch AWS credentials for the role using the EC2 IMDS endpoint. The fetched credentials are then used to sign the GetCallerIdentity request and generate the subject_token value. The IMDS endpoint is also used to retrieve the AWS region and plug it in the regional_cred_verification_url parameter. If your AWS workload is an EC2 instance, you are all set.

However, if you are on ECS, then the client auth libraries will not work as expected. ECS tasks fetch their task role credentials from the endpoint: 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI. And since the credential_source configuration is hardcoded by GCP to have EC2 IMDS endpoint, using the GCP client libraries for WIF from ECS tasks fails. I am also not the first one to complain about this:

https://github.com/googleapis/google-auth-library-php/issues/496 (Nov 2023)
https://github.com/googleapis/google-auth-library-java/pull/1374 (Mar 2024)

Solution and Workaround

So what can you do to make GCP WIF work with AWS ECS? GCP client libraries do provide a way to deal with this. You can define a custom AWS credential supplier to determine how you want to retrieve AWS credentials in the workload. This requires defining two functions (get AWS region and get AWS credentials), telling GCP client library how it should fetch the AWS credentials for the role and the region (for regional_cred_verification_url parameter). Here are the class structures for Java and Python libraries:

For the google-auth Java library: Using a custom supplier with AWS
For the google-auth Python library: class AwsSecurityCredentialsSupplier

Defining a custom AWS credential supplier is not very complicated. But surprisingly there is a dearth of standard examples. I have put together a Python implementation here: GCP-WIF-AwsSecurityCredentialsSupplier. The repository also contains Terraform code to setup the necessary configuration in GCP and AWS and test out the implementation. Here is an excerpt of the implementation:

class CustomAwsSecurityCredentialsSupplier(aws.AwsSecurityCredentialsSupplier):
   def get_aws_security_credentials(self, context, request) -> aws.AwsSecurityCredentials:
        try:
            # AWS SDKs automatically fetch credentials using ECS Metadata endpoint
            # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/security-iam-roles.html#security-iam-task-role
            # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.get_credentials
            session = boto3.Session()
            credentials = session.get_credentials()
            return aws.AwsSecurityCredentials(credentials.access_key, credentials.secret_key, credentials.token)
        except Exception as e:
            logger.error("Error fetching AWS credentials: %s", e)
            raise google.auth.exceptions.RefreshError(e, retryable=True)

    def get_aws_region(self, context, request) -> str:
        try:
            # ECS tasks have AWS_REGION environment variable set
            # https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-environment-variables.html
            if os.environ.get("AWS_REGION"):
                return os.environ.get("AWS_REGION")
            else:
                raise ValueError("AWS_REGION environment variable is not set")
        except Exception as e:
            logger.error("Error fetching AWS region: %s", e)
            raise google.auth.exceptions.RefreshError(e, retryable=True)

Workaround

Such custom AWS credential suppliers do not work well with gcloud CLI. Also, there could be situations where implementing a custom AWS credential supplier isn’t the most feasible option. For example, if you aren’t calling the google-auth library directly (being called internally by another package), you cannot pass a custom class to it.

Due to these limitations, we need a workaround to enable the auth libraries/gcloud to work automatically with GCP’s credential configuration file. The problem remains though that this file has hardcoded EC2-based credential_source values and that won’t work for ECS. So we need a way to bypass this credential_source configuration altogether.

We can leverage the design of the GCP google-auth client libraries to our advantage. These libraries, by default, look for AWS credentials in the environment variables first ³ ⁴. If they do not find AWS credentials there, they then use the EC2 IMDS endpoint specified in credential_source to fetch the credentials.

Excerpt from google-auth Python library:

def get_aws_security_credentials(self, context, request):

    # Check environment variables for permanent credentials first.
    # https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html
    env_aws_access_key_id = os.environ.get(environment_vars.AWS_ACCESS_KEY_ID)
    env_aws_secret_access_key = os.environ.get(
        environment_vars.AWS_SECRET_ACCESS_KEY
    )
    # This is normally not available for permanent credentials.
    env_aws_session_token = os.environ.get(environment_vars.AWS_SESSION_TOKEN)
    if env_aws_access_key_id and env_aws_secret_access_key:
        return AwsSecurityCredentials(
            env_aws_access_key_id, env_aws_secret_access_key, env_aws_session_token
        )

    imdsv2_session_token = self._get_imdsv2_session_token(request)
    role_name = self._get_metadata_role_name(request, imdsv2_session_token)

    # Get security credentials.
    credentials = self._get_metadata_security_credentials(
        request, role_name, imdsv2_session_token
    )

    return AwsSecurityCredentials(
        credentials.get("AccessKeyId"),
        credentials.get("SecretAccessKey"),
        credentials.get("Token"),
    )

Excerpt from google-auth Java library:

public AwsSecurityCredentials getCredentials(ExternalAccountSupplierContext context)
    throws IOException {
  // Check environment variables for credentials first.
  if (canRetrieveSecurityCredentialsFromEnvironment()) {
    String accessKeyId = environmentProvider.getEnv(AWS_ACCESS_KEY_ID);
    String secretAccessKey = environmentProvider.getEnv(AWS_SECRET_ACCESS_KEY);
    String token = environmentProvider.getEnv(AWS_SESSION_TOKEN);
    return new AwsSecurityCredentials(accessKeyId, secretAccessKey, token);
  }

  Map<String, Object> metadataRequestHeaders = createMetadataRequestHeaders(awsCredentialSource);

  // Credentials not retrievable from environment variables - call metadata server.
  // Retrieve the IAM role that is attached to the VM. This is required to retrieve the AWS
  // security credentials.
  if (awsCredentialSource.url == null || awsCredentialSource.url.isEmpty()) {
    throw new IOException(
        "Unable to determine the AWS IAM role name. The credential source does not contain the"
            + " url field.");
  }
  String roleName = retrieveResource(awsCredentialSource.url, "IAM role", metadataRequestHeaders);

  // Retrieve the AWS security credentials by calling the endpoint specified by the credential
  // source.
  String awsCredentials =
      retrieveResource(
          awsCredentialSource.url + "/" + roleName, "credentials", metadataRequestHeaders);

  JsonParser parser = OAuth2Utils.JSON_FACTORY.createJsonParser(awsCredentials);
  GenericJson genericJson = parser.parseAndClose(GenericJson.class);

  String accessKeyId = (String) genericJson.get("AccessKeyId");
  String secretAccessKey = (String) genericJson.get("SecretAccessKey");
  String token = (String) genericJson.get("Token");

  // These credentials last for a few hours - we may consider caching these in the
  // future.
  return new AwsSecurityCredentials(accessKeyId, secretAccessKey, token);
}

So if you set the environment variables for your AWS credentials, the libraries will use them directly. This eliminates the need of custom AWS credential supplier and also bypasses the credential_source configuration in the GCP credential configuration file.

Getting this to work is as simple as executing a bash script upon ECS task setup to set the environment variables:

#!/usr/bin/env bash
set -euo pipefail

if [[ -z "${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI:-}" ]]; then
  echo "ERROR: AWS_CONTAINER_CREDENTIALS_RELATIVE_URI is not set." >&2
  exit 1
fi

url="http://169.254.170.2${AWS_CONTAINER_CREDENTIALS_RELATIVE_URI}"

# Fetch credentials
response=$(curl --fail --silent --show-error "$url")

export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export AWS_SESSION_TOKEN

AWS_ACCESS_KEY_ID=$(jq -r '.AccessKeyId' <<< "$response")
AWS_SECRET_ACCESS_KEY=$(jq -r '.SecretAccessKey' <<< "$response")
AWS_SESSION_TOKEN=$(jq -r '.Token' <<< "$response")

unset response

It is important to note one caveat with this workaround: credentials fetched for ECS tasks are valid for 6 hours. So if your ECS task needs longer GCP access, you would need a refresh mechanism in place.

Closing Thoughts

To perform GCP Workload Identity Federation with AWS ECS tasks, one needs to either implement a custom AWS credential supplier or set AWS credentials in environment variables upon task setup. While there isn’t a considerable overhead to doing either of these manually, it would be nice to have in-built support in google-auth client libraries (something something about having paved paths).

Additionally, a dearth of standard examples of how to implement custom AWS credential suppliers, increases the chances of erroneous developer implementations.