Most multi-region Kubernetes guides stop at the YAML. This one goes through the YAML, the failover test, and the actual cost at the end of the month. We'll deploy a stateless web tier across three of our regions — IAD, FRA, NRT — with traffic steering and a regional Postgres replica.
Topology
Three control planes, three data planes, one global service that routes via our anycast load balancer. Postgres is primary in IAD, with read replicas in FRA and NRT. Writes always go to IAD; reads can serve from anywhere.
The cluster manifest
apiVersion: rsh.cloud/v1
kind: KubernetesCluster
metadata:
name: web-global
spec:
regions:
- name: iad
nodes: 3
size: vps-4
- name: fra
nodes: 3
size: vps-4
- name: nrt
nodes: 2
size: vps-4
loadBalancer:
type: anycast
healthCheck:
path: /healthz
intervalSeconds: 5The failover test
We tear down the IAD region while load is running. Within 18 seconds, anycast withdraws the IAD prefix, traffic shifts to FRA and NRT, and Postgres writes start failing because the primary is gone. We promote the FRA replica. Writes recover at the 90-second mark.
Postgres failover is the slowest part of any geographically-distributed setup. Plan for at least 60 seconds of write unavailability during a real region loss.
What it costs
Three regions, eight nodes, anycast LB, three Postgres instances, cross-region replication: about $480/month. Compare to running this on one of the big three clouds with the same redundancy and you're looking at roughly $1,400. The savings come mostly from not paying egress between regions.