Securing Load balanced Authenticated Cloud Run Services using IAP

When designing your cloud run services, you should always consider following points

1. How to run same service in multiple region for multi-regional support
2. You should think about securing cloud run service with authentication support so that only authenticated users can invoke your cloud run service

Both of these features are well documented on Google cloud documentation and can be just implemented by following Google cloud documentation.

But we should remember following points

1. When using Authenticated Cloud run either as (PubSub push url where we configure audience, although this is optional) , JWT audience for Cloud Run (and GCF) bound ID tokens is region specific. That means if your cloud run (say us-west and us-central) is load balanced, the authentication may not work as expected because the JWT token is region bound and this token is generated by PubSub and will eventually fail if the request lands on the other region cloud run service because the token is generated by PubSub while which cloud run to use is being decided by the Load Balancer.
2. The authentication of the JWT token in performed at the Cloud run end, The token is verifies based on the below rules.

• If the used behind the load balancer and audience is matching the cloud run service url , then authentication is performed normally other 403 error is raised
• If used cloud run service url directly then audience is optional and JWT token validation happens normally.

To solve this problem, I propose below architecture

Cloud Run Secured by Cloud IAP

Below is the thought process on this architecture

1. Cloud run is deployed in 2 different region as Authenticated and Ingress with Internal and Load Balancer only
2. These cloud run are load balanced by HTTPS Load Balancer
3. Cloud IAP is deployed on top of Load Balancer so that only those users added as Cloud IAP WebApp User, can invoke the the cloud IAP
4. Now Since cloud is authenticated one, Cloud IAP will not be able to invoke the Cloud Run.
5. To Solve this, we need to give cloud run invoker permission to Cloud IAP
6. But How to give this permission to Cloud IAP?

This is an unsolved and undocumented problem on Google Cloud Platform.

When you configure a IAP on top of Load Balancer/App Engine/Managed Instance Group(MIG) on GCP, GCP created oAuth credentials behind the scene for you. These credentials are with name IAP-{LOAD_BALANCER_NAME} or IAP-{APP_ENGINE_SERVICE_NAME}

The above line is not documented anywhere on GCP or at-least I could not find it on GCP.

Now to give Cloud IAP permission to invoke cloud run service, perform below steps

1. Go to credentials page
2. Copy the Client id from the respective OAuth 2.0 Client ids section for you Cloud IAP

3. Now give Cloud Run invoker permission to this Client Id.

Now to allow users to invoke or authenticate though the Cloud IAP , you must give them IAP WebApp User permission. This way you can expose your cloud run service without giving end users Cloud Run invoker role. You just add them to Cloud IAP.

Solving the same for PubSub Push delivery for multi regional service.

Workflow is as below.

1. Create a push subscription with Push url pointing to LB domain
2. Service a service account with push url which have IAP WebApp User Permission
3. In the audience field, enter IAP client ID to which you give cloud run invoker permission in earlier permission.

Hope you will like this unsolved and undocumented problem for Cloud Run and Load Balancer.