OpenTelemetry Collector deep dive

The pipeline model

receivers ──► processors ──► exporters
 (otlp,        (memory_limiter,  (otlp → Tempo,
  prometheus,   k8sattributes,    prometheusremotewrite → Prometheus/Mimir,
  filelog…)     filter, batch…)   otlphttp → Loki…)

Each pipeline handles one signal (traces, metrics or logs). A component can be used in several pipelines; connectors act as an exporter for one pipeline and a receiver for another.

A working config

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317      # recent versions default to localhost; bind explicitly in pods
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  k8sattributes:
    extract:
      metadata: [ k8s.namespace.name, k8s.pod.name, k8s.deployment.name, k8s.node.name ]
  resource:
    attributes:
      - key: k8s.cluster.name
        value: prod-eu-1
        action: upsert
  batch: {}

exporters:
  otlp/tempo:
    endpoint: tempo.observability.svc:4317
    tls:
      insecure: true                 # lab only; use TLS in production
  prometheusremotewrite:
    endpoint: http://prometheus.monitoring.svc:9090/api/v1/write
  otlphttp/loki:
    endpoint: http://loki-gateway.observability.svc/otlp
  debug: {}

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ memory_limiter, k8sattributes, resource, batch ]
      exporters: [ otlp/tempo ]
    metrics:
      receivers: [ otlp ]
      processors: [ memory_limiter, k8sattributes, resource, batch ]
      exporters: [ prometheusremotewrite ]
    logs:
      receivers: [ otlp ]
      processors: [ memory_limiter, k8sattributes, resource, batch ]
      exporters: [ otlphttp/loki ]

Notes:

• Prometheus accepts remote write only with --web.enable-remote-write-receiver; recent Prometheus versions can also ingest OTLP directly (--web.enable-otlp-receiver).
• Loki 3.x ingests OTLP natively at /otlp; the older Loki-specific exporter is deprecated.
• k8sattributes needs RBAC to read pods, namespaces and ReplicaSets.
• Components live in different distributions: the core otelcol has few; otelcol-contrib has most (k8sattributes, tail_sampling, filelog…); you can also build your own with the OpenTelemetry Collector Builder.

Processors you'll use

Agents, gateways and the Operator

• Agent (DaemonSet): apps send OTLP to the node-local Collector ($(NODE_IP):4317 via the downward API); it also scrapes host and kubelet metrics and tails log files.
• Gateway (Deployment + HPA): central processing, tail sampling, routing to tenants/backends, credentials for backends in one place.
• The OpenTelemetry Operator manages Collectors as OpenTelemetryCollector resources (modes: deployment, daemonset, statefulset, sidecar) and auto-instrumentation through Instrumentation resources plus pod annotations such as instrumentation.opentelemetry.io/inject-java: "true".

Watching the Collector

• Collector self-metrics (port 8888 by default): accepted, refused and failed-to-send items per component, queue sizes. Alert on refused and failed sends.
• debug exporter prints data flowing through: invaluable for "why is this attribute missing?", but remove it afterwards (it's noisy).
• otelcol-contrib validate --config catches config errors before a rollout.

Recap

• Receivers → processors → exporters, per signal pipeline; connectors link pipelines.
• Always memory_limiter first, k8sattributes for context, batch for efficiency.
• Agent (DaemonSet) + gateway (Deployment) patterns; the Operator manages Collectors and auto-instrumentation.
• Watch the Collector's own metrics; debug with the debug exporter; validate configs before rollout.