Observability: Cloud Logging, Cloud Monitoring & Managed Prometheus

What you get by default

GKE's dashboards in the console (cluster, workload, node views) build on these.

Your metrics: Managed Service for Prometheus

Apps expose /metrics as usual. With managed collection enabled, you declare what to scrape:

apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: shop
  namespace: shop
spec:
  selector:
    matchLabels: { app: shop }
  endpoints:
    - port: metrics
      interval: 30s

• ClusterPodMonitoring does the same across namespaces; Rules / ClusterRules hold recording and alerting rules in Prometheus format.
• Query with PromQL in Cloud Monitoring, or point Grafana at it (Google provides a data-source syncer for authentication).
• Data is stored by Google with long retention and global querying across clusters and projects (metrics scopes).
• Self-deployed collection (your own Prometheus that writes to the managed backend) is an option when you need full Prometheus Operator compatibility.
• Cost follows samples ingested: watch high-cardinality labels and scrape intervals.

Traces

Instrument apps with OpenTelemetry and export to Cloud Trace (directly or through an OpenTelemetry Collector in the cluster, using Workload Identity for credentials). Load balancer logs and trace IDs let you follow a request from the edge into the app. The Observability with OpenTelemetry track covers instrumentation in depth.

Alerts and SLOs

• Alert on symptoms users feel: error rate, latency, saturation of the critical path, SLO burn rate.
• Cloud Monitoring supports SLOs on request-based or windows-based indicators, with burn-rate alert policies.
• Platform alerts worth having from day one:

Route alerts to the owning team through notification channels (PagerDuty, Slack, email), with runbook links in the alert documentation.

Keeping logging affordable

Ingestion is usually the biggest observability cost. In order:

1. Reduce at the source: app log levels, no per-request debug logs in production.
2. Exclusion filters on the _Default sink for noisy, low-value logs (health checks, chatty sidecars).
3. Sinks to route logs: security logs to a long-retention bucket or BigQuery, the rest with short retention.
4. Sample access logs at the load balancer if full logs aren't required.

gcloud logging sinks update _Default \
  --add-exclusion=name=healthchecks,filter='resource.type="k8s_container" AND httpRequest.requestUrl:"/healthz"'

Recap

• GKE collects system and workload logs and system metrics by default; control-plane and kube-state metrics are optional packages.
• Managed Service for Prometheus: you declare PodMonitoring, Google stores and serves PromQL.
• Traces with OpenTelemetry to Cloud Trace.
• Alert on symptoms and SLO burn rate, with platform alerts for pending Pods, NAT, disks and the edge.
• Control cost with log levels, exclusions, sinks and retention.