gRPC & Protobuf · Prasanta Kumar Behera

gRPC is everywhere in cloud-native infrastructure. Learn how it works (Protocol Buffers over HTTP/2), its streaming modes and status codes, why it load-balances badly by default in Kubernetes, and how to fix and debug it.

What gRPC is

gRPC is a remote procedure call framework: you define a service's methods and messages in a .proto file, generate client and server code in many languages, and call remote methods like local functions. Underneath it uses Protocol Buffers (compact binary messages) over HTTP/2.

Kubernetes itself uses it (CRI between kubelet and containerd, etcd's API, CSI and device plugins), and so do many services you'll run.

Normal web APIs are like sending letters in plain language. gRPC is like two friends who agreed on a secret shorthand (protobuf) and keep a walkie-talkie line open all day (HTTP/2): super fast, many messages at once. But if you add more friends later, the old walkie-talkie line is still tuned to the same person, so the new friends sit around with nothing to do.

syntax = "proto3";
package orders.v1;

service Orders {
  rpc GetOrder(GetOrderRequest) returns (Order);                 // unary
  rpc WatchOrders(WatchRequest) returns (stream OrderEvent);     // server streaming
}

message GetOrderRequest { string id = 1; }
message Order { string id = 1; string item = 2; int32 quantity = 3; }
message WatchRequest { string customer = 1; }
message OrderEvent { Order order = 1; string type = 2; }

Four call types

Type	Shape	Example
Unary	request → response	Get an order
Server streaming	request → stream of responses	Watch events
Client streaming	stream of requests → response	Upload metrics
Bidirectional	stream ↔ stream	Chat, live sync

Status codes, deadlines, retries

gRPC has its own status codes (sent in HTTP/2 trailers, not the HTTP status):

Code	Meaning	Retry?
`OK`	Success	—
`INVALID_ARGUMENT`, `NOT_FOUND`, `PERMISSION_DENIED`	Caller's problem	No
`DEADLINE_EXCEEDED`	Took longer than the caller's deadline	Maybe, carefully
`UNAVAILABLE`	Server down, overloaded or connection lost	Yes, with backoff
`RESOURCE_EXHAUSTED`	Quota or rate limit	Later, with backoff

Always set deadlines. They propagate through chains of calls, so a slow dependency can't hold resources forever.

The Kubernetes load-balancing trap

A normal ClusterIP Service balances connections (L4). gRPC clients open one long-lived HTTP/2 connection and send every request over it. Result: after scaling up, new pods sit idle; one busy client can overload one pod.

Fixes, from simplest to most powerful:

Approach	How
L7 proxy / service mesh	Envoy, Linkerd, Istio or a gRPC-aware gateway balance per request
Client-side balancing	Headless Service (`clusterIP: None`) + the gRPC client's `round_robin` policy with DNS re-resolution
Bounded connection age	Server option `MaxConnectionAge` makes clients reconnect periodically, spreading load again

Health checks

gRPC defines a standard health service (grpc.health.v1.Health), and Kubernetes supports it natively:

readinessProbe:
  grpc:
    port: 50051
livenessProbe:
  grpc:
    port: 50051
  initialDelaySeconds: 10

Debugging gRPC

$ kubectl port-forward svc/orders 50051:50051
$ grpcurl -plaintext localhost:50051 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
orders.v1.Orders
$ grpcurl -plaintext -d '{"id": "42"}' localhost:50051 orders.v1.Orders/GetOrder
{
  "id": "42",
  "item": "pizza",
  "quantity": 2
}

list and describe need server reflection enabled; otherwise pass the .proto files with -proto. Through TLS ingresses, drop -plaintext and make sure every hop speaks HTTP/2 (lesson 10).

Try it: see the imbalance, then fix it

Use any small gRPC demo server that reports its hostname in responses (or write one in Python with grpcio following the gRPC quick start).

Deploy 2 replicas behind a normal ClusterIP Service; call it in a loop from one client pod and count responses per pod.
Scale to 6 replicas; keep the same client running. Do the new pods receive traffic?
Switch to a headless Service and enable round_robin in the client (in gRPC's service config, or via client options). Count again.
Add gRPC readiness and liveness probes and check they pass.

Going deeper: gRPC in production

Keep .proto files versioned (orders.v1), make only backward-compatible changes (add fields, never reuse numbers), and lint them with buf in CI.
Watch message sizes (default limits of a few MB), stream lifetimes, and HTTP/2 keepalive settings on both clients and proxies. Idle timeouts on load balancers break streams silently.
For browsers, gRPC-Web or Connect protocols bridge HTTP/1.1 and HTTP/2 limitations.
Meshes give gRPC-aware retries, timeouts and per-method metrics without code changes (see Service Mesh).

Recap

gRPC = Protocol Buffers over HTTP/2; unary and streaming calls.
Its own status codes; always set deadlines; retry only retryable codes with backoff.
ClusterIP balances connections, so use an L7 proxy/mesh, client-side balancing with a headless Service, or max connection age.
Native gRPC probes; debug with grpcurl (reflection).

This site is a public version of my personal engineering knowledge hub. It intentionally excludes confidential company information and internal operational details.