“How should I expose my applications outside of my GKE cluster?”

I recently reviewed the options for GKE ingress and now I remember how overwhelming the topic may be for those that are new to kubernetes and/or Google Cloud. There are multiple options without a consistently clear winner and the best path depends on the use case. When considering options to expose your k8s application(s), here are some questions to reflect upon:

Do I want to expose a single service or many services?
Do I need to expose all of my service(s) with a single IP address?
Do I need to expose all of my service(s) with a single hostname?
Do I care if it uses 1 load balancer or multiple load balancers?
Do I need the ability to rewrite headers or paths in requests?
Do my services run in a single namespace or across many namespaces?
Is routing my traffic efficiently and with minimal latency critical for my application?
Would I prefer to run and manage my own load balancing resources or should Google be responsible for the security, upgrades, and scaling of resources based on demand?
Do I need to protect my endpoint from threats such as DDoS or other layer 7 attacks?
Do I need more advanced features such as rate limiting, accounting api chaining, or bill back?

For reference, here’s the current document that I use as a starting point for my refresher.

https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer

On the left navigation pane, there are 4 high level options for exposing services outside the cluster under the highlighted index “Load balance traffic” in the image below

High level options with my thoughts on level of effort and amount of flexibility are below

Methods of exposing a service outside the cluster

Load Balancer Service. (Easiest, least flexible)
GKE Ingress (Easy/Medium effort, some flexibility)
GKE Gateway. (Medium effort, some flexibility)
3rd Party Ingress- NGINX or other. (Easy/Medium effort, some flexibility)
API Gateway or Service Mesh- Apigee, Anthos Service Mesh, Istio, other. (Requires additional setup, maximum flexibility)

Load Balancer Services

These are built into GKE. Using them is as simple as ensuring that your GKE service is set to `type: LoadBalancer` which will auto provision a GCE load balancer that’s dedicated to accept external traffic for this service.

Pros:

Easy to set up
Easy to manage (there are not many additional configuration parameters)
GCP Load Balancer is managed and scaled by Google (upgrades, scaling, etc)
Support to add on Google Cloud Armor (DDos and application attack protection)

Cons:

Load Balancer dedicated for each service (unique IP address, hostname)
Doesn’t easily allow for division of responsibilities (InfoSec may require traffic enter at a trusted point for security purposes)
No additional configuration parameters to alter path, or headers
HTTPS/TLS Certificate must be configured in your application

Additional Doc- https://cloud.google.com/kubernetes-engine/docs/concepts/service-load-balancer

GKE Ingress

Supports NodePort load balancing by default, but it also supports auto-enabling the preferred method, container native load balancing, if certain preconditions are met. Container native load balancing is preferred because it reduces the number of hops in the traffic path.

With the NodePort method of load balancing (on the left above), traffic is initially routed to different GKE nodes through an instance group on the allocated NodePort. The GKE node that receives the traffic then needs to reference iptables to translate the packet IP address to match one of the pods behind the kubernetes service. When the pod responds to the request, traffic is routed back to the GKE node where the traffic is translated via iptables, it’s then translated back and the node responds to the request.

In the right image with the VPC Native clusters, which means pods are allocated routable IP addresses, the load balancer is able to route directly to the pods behind the service, which saves us a hop and resources because there’s no need for iptables translations.

Configuration for both options with GKE ingress is managed by installing the GKE Ingress controller and exposing services with a Kubernetes Ingress object type. The ingress object type gives additional control over routing by hostname, path, and use of hostname wildcards within a single namespace. Ingress objects on GKE do not support cross namespace routing therefore all the services must be in a single namespace We’re unable to have the gateways in a namespace with the applications in a different namespace with this path. Aso note that creating an Ingress object creates a load balancer so this is a 1 to 1 mapping.

Pros:

Relatively easy to set up
Additional routing flexibility (virtual hostnames, path)
Very efficient routing paths- fewer hops, fewer resources utilized
GCP Load Balancer is managed and scaled by Google (upgrades, scaling, etc)
Support to add on Google Cloud Armor (DDoS and application attack protection)
Native support for HTTPS/SSL certificate configuration

Cons:

A single load balancer can only route to services in a single namespace
This approach has limited flexibility on division of responsibility since Ingress controls the provisioning of the load balancer and the application routing
Combining multiple Ingress objects into a single load balancer is not supported
Multiple load balancers are required for many environments

Additional Info- https://cloud.google.com/kubernetes-engine/docs/concepts/ingress

GKE Gateway API

This is the direction that Google and the industry is moving because it adds additional flexibility with division of responsibility such as allowing an Operations &/or InfoSec team to manage the cluster entry point (gateway) configuration while allowing developers to handle routing configuration for their apps. Note that GKE Gateway is currently in preview status (not GA/Generally Available) but is expected to be GA later this year.

Gateway’s an exciting option that’s designed to split the roles of gateway operator and application routing but it also introduces the ability to route across namespaces. This for example means that gateways can be provisioned in a namespace such as ‘gateways’ while applications can be distributed across other namespaces.

This path looks promising but through my testing I see there are some caveats for many customers until it’s GA and it has a few additional key features.

I test deploying an application into it’s own namespace and then configure an HTTPRoute for my service. I then test the request through the load balancer and receive a 404. Oh no! What’s happening? After digging deeper I learn that the gateway forwards my prefix to the service and my application doesn’t have a document here.

To illustrate with an example:

curl my.test.service.com/app1     #-> This app works if it listens and responds to /* (all paths)

curl my.test.service.com/app2  #-> This app won’t work if it only handles / (no wildcard) because there is not /app2

This sent me scouring the Google docs, which didn’t turn up anything about how to rewrite the header, then to the public docs which brings us this link.

https://gateway-api.sigs.k8s.io/v1alpha2/guides/http-redirect-rewrite/

Upon further research, I see that the rewrite and redirect capabilities are still in alpha phase and Google doesn’t include alpha when shipping Gateway so keep an eye out for GA and when these features make it into the product if you need them.

Pros:

Relatively easy to set up
Some support for division of responsibility
Supports a single load balancer with multiple cross namespace services behind it
GCP Load Balancer is managed and scaled by Google (upgrades, scaling, etc)
Support to add on Google Cloud Armor (DDoS and application attack protection)
Native support for HTTPS/SSL certificate configuration

Cons:

Preview phase- no GA support yet (expected later this year- stay tuned!)
Does not yet include header rewrite capabilities which would usually require introducing additional technologies such as a service mesh or manual management of load balancer components
Slight learning curve with new k8s objects

Additional Info- https://cloud.google.com/kubernetes-engine/docs/concepts/gateway-api

NGINX Ingress

It’s been around for a while and is well supported by the community. It’s incredibly powerful and works with default Kubernetes objects such as Deployment, Service, and Ingress. It does support exposing multiple services through a single load balancer and also supports merging multiple ingress objects into a single ingress gateway. NGINX Ingress also supports header rewriting via annotation on Ingress objects.

Pros:

Not difficult to set up but it is more work with the additional capabilities
Strong support for division of responsibility
Supports a single load balancer with multiple cross namespace services behind it
Well supported within the open source community
Paid enterprise version and support available from NGINX
Support to add on Google Cloud Armor (DDoS and application attack protection)
Native support for HTTPS/SSL certificate configuration

Cons:

Not supported directly by Google
Similar to NodePort load balancing as described under GKE Ingress above, an additional hop and iptables translation is required in the communication path
NGINX nodes run in cluster and are managed by customers (security, upgrades, scaling, etc)

Additional Info: https://cloud.google.com/community/tutorials/nginx-ingress-gke

API Gateway and/or Service Mesh technologies

API Gateways, such as Apigee warrant their own blogs and write ups. They will require some up front design and implementation work but provide significant capabilities on top of previous options. A few of the features Apigee provides:

Rate limiting by bandwidth and or number of requests
Ability to monetize your applications
API chaining
Standardize user experience across disparate apis
Simplify integration of security controls

Service mesh technologies such as Anthos Service Mesh (ASM) also introduce advanced features such as:

Automation of certificate management/rotation
Simple implementation of mTLS between services
Distributed application tracing
Standardized metrics collection with ability to define SLO’s
Extensive routing controls such as A/B Testing, Blue Green Deployments
Decouple availability logic such as retry logic and circuit breaking
Fault Injection for testing purposes

These topics are too complex to dig into here but the technologies are likely to be essential as your cluster grows. Keep an eye out for future posts that dig more into these.

Pros:

Maximum flexibility
Introduces many additional features beyond exposing an application

Cons:

Additional cost
More design and planning time to get it set up

Additional Info-

ASM- https://cloud.google.com/anthos/service-mesh

Apigee- https://cloud.google.com/apigee

UPDATE July 10 2023– Google just announced that URL Rewrites and User Request Response headers are now supported. At this point, I suggest exploring use of the Gateway API over GKE Ingress. Unless you have some edge cases there’s a good chance it will meet your needs

New Features for This Release:

New GatewayClasses supporting the regional external Application Load Balancer (only available with GKE Gateway!)
URL Rewrites and Path Redirects (only available with GKE Gateway!)
Identity-aware Proxy (IAP) Integration
Custom request and response headers

https://cloud.google.com/kubernetes-engine/docs/how-to/gatewayclass-capabilities