Production GKE Platform — From Zero to Production›03 · Organisation, projects & Shared VPC

Lesson 03 of 18 · Part 2 — Build the platform

Organisation, projects & Shared VPC

Lay the ground a GKE platform stands on: folders and projects per environment, a Shared VPC run by the network team, subnets with secondary ranges, Cloud NAT and Private Google Access for private nodes, firewall rules, and the IAM that lets a service project's clusters use the host's network.

Practitioner
Key wordsresource hierarchyfoldersprojectsShared VPChost projectservice projectsubnetssecondary rangesCloud NATCloud RouterPrivate Google Accessfirewall rulesorganization policies

Start with the hierarchy

Organization (example.com)
├── Folder: platform
│   ├── net-host-prod        Shared VPC host (prod)
│   ├── net-host-nonprod     Shared VPC host (dev, staging)
│   └── platform-shared      Artifact Registry, CI identities, DNS
├── Folder: prod
│   ├── gke-prod             clusters (service project of net-host-prod)
│   └── data-prod            databases, buckets
└── Folder: nonprod
    ├── gke-dev, gke-staging
    └── data-nonprod

Projects are the blast-radius, IAM, quota and billing boundary. Separate prod from non-prod by project (and usually by folder), so a mistake or a compromised credential in dev can't touch production. Organization policies set at the folder level apply to every project below.

The organisation is a company campus, folders are buildings, projects are offices. The network team owns the corridors and cabling (Shared VPC) for a whole building; each office team just plugs in, without being able to rewire the corridor.

Shared VPC

Piece Owned by Holds
Host project Network team The VPC, subnets, routes, Cloud NAT, firewall policies, VPN/Interconnect
Service project Platform or app team GKE clusters, VMs, load balancers that use the host's subnets

What GKE in a service project needs, in the host project:

  • The service project's GKE service agent (service-<PROJECT_NUMBER>@container-engine-robot.iam.gserviceaccount.com) gets the Host Service Agent User role on the host project.
  • The GKE service agent and the Google APIs service agent of the service project get Compute Network User on the subnets the cluster uses (subnet-level grants keep teams in their own subnets).
  • Firewall rules: GKE wants to create rules for its control plane, health checks and load balancers. Either give it the Security Admin role in the host (simplest), or let the network team create those rules and accept that GKE reports what it couldn't create.

Subnets for GKE

A GKE subnet has three ranges (sizes in lesson 04):

gcloud compute networks subnets create gke-prod-ew1 \
  --project net-host-prod --network prod-vpc --region europe-west1 \
  --range 10.10.0.0/22 \
  --secondary-range pods=10.20.0.0/16,services=10.30.0.0/20 \
  --enable-private-ip-google-access
  • Primary range: node IPs (and internal load balancers).
  • Secondary range for Pods, secondary range for Services: used by the VPC-native cluster.
  • One subnet per region per environment is a common start; a separate subnet per cluster keeps IP plans and firewall rules cleaner when you run several clusters.

Egress: Cloud NAT and Private Google Access

Production nodes are private (no external IPs). They still need:

Destination How
Google APIs (Artifact Registry, Logging, Monitoring, Secret Manager) Private Google Access on the subnet, or Private Service Connect endpoints
The internet (public registries, external APIs) Cloud NAT on a Cloud Router in that region
On-prem Cloud VPN or Interconnect in the host project

Size Cloud NAT for busy clusters: each NAT IP has a fixed number of source ports, and many connections to the same destination can exhaust them. Watch the NAT dropped-packets metrics, and use dynamic port allocation or more NAT IPs if needed.

Firewall rules

  • GKE creates the rules it needs (control plane to nodes, node-to-node, health checks for load balancers).
  • Add your own as hierarchical firewall policies (org or folder) and network firewall policies for broad rules (deny everything from the internet except the load balancers' ranges), and target node rules by service account rather than network tags where you can.
  • Inside the cluster, network policy controls Pod-to-Pod traffic (lesson 04); VPC firewall rules don't see individual Pods unless you design for it.

Organization policies worth setting early

  • Allowed regions for resources.
  • No external IPs on VMs (private nodes everywhere).
  • Restrict which projects may use Shared VPC subnets.
  • Require OS Login, disable default service-account grants, require CMEK where compliance needs it.

Test policies on a non-prod folder first: some block GKE operations in non-obvious ways (for example a policy forbidding external IPs also blocks public node pools).

Try it: a Shared VPC for GKE (two projects in a sandbox organisation)

  1. Create a host project and a service project; enable the Compute and Container APIs in both.
  2. Enable Shared VPC on the host, create a custom VPC and a subnet with pods and services secondary ranges and Private Google Access.
  3. Attach the service project and grant its GKE service agent the Host Service Agent User role and Compute Network User on the subnet.
  4. Create Cloud NAT in the host's region.
  5. In the service project, create a private Standard cluster using the shared subnet and its secondary ranges (lesson 05 shows the flags). Pull a public image to prove NAT works.

Going deeper: landing zones

  • Build the hierarchy, Shared VPCs, NAT and org policies with Terraform (Google's Cloud Foundation Fabric or the terraform-example-foundation are good references), before any cluster exists.
  • Plan IP ranges for the whole organisation at once, including on-prem and other clouds, so hybrid links never collide (lesson 04).
  • Use VPC Service Controls perimeters around sensitive data projects to limit data exfiltration through Google APIs; test them carefully with GKE, Artifact Registry and logging.

Recap

  • Organization → folders → projects; projects separate prod from non-prod.
  • Shared VPC: the network team owns the host project; clusters live in service projects with subnet-level access.
  • GKE subnets have a primary range for nodes and secondary ranges for Pods and Services.
  • Private nodes need Private Google Access for Google APIs and Cloud NAT for the internet.
  • Set organization policies and firewall policies early, and test them in non-prod.

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.