Distributed Tracing with OpenTelemetry

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                        Trace Flow                                    │
│                                                                     │
│  Demo UI → Backend (FastAPI + OTEL) → Kafka Broker (+ OTEL Agent)  │
│                │                            │                       │
│       OTLP spans (gRPC)             OTLP spans (gRPC)              │
│                ↓                            ↓                       │
│            OTEL Collector  ←────────────────┘                       │
│                ↓                                                    │
│            Jaeger v2 (UI on :16686)                                 │
└─────────────────────────────────────────────────────────────────────┘

Components

OTEL Collector (k8s/observability/otel-collector.yaml)

• Receives traces via OTLP gRPC (:4317) and HTTP (:4318), plus Zipkin (:9411)
• Exports to Jaeger via OTLP gRPC
• Debug exporter enabled (logs spans to stdout for troubleshooting)

Jaeger v2 (k8s/observability/jaeger.yaml)

• All-in-one deployment (in-memory storage for PoC)
• Image: jaegertracing/jaeger:latest (⚠️ tag 2 does not exist!)
• Exposes OTLP gRPC (:4317), OTLP HTTP (:4318), UI (:16686), Zipkin (:9411)

Network Policy (k8s/observability/network-policy.yaml)

• Critical: Calico is deny-all by default (even intra-namespace!)
• Allows: intra-namespace (podSelector: {}), from knative-eventing, from default
• Ports: 4317, 4318, 9411, 16686

---

⚠️ KNative Eventing Kafka Broker — Tracing Setup

This is the hard-won knowledge. Read carefully.

What DOESN'T work (despite what docs say)

The official KNative docs say to configure config-observability ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-observability
  namespace: knative-eventing
data:
  tracing-protocol: http/protobuf
  tracing-endpoint: http://otel-collector.observability.svc.cluster.local:4318/v1/traces
  tracing-sampling-rate: "1"

This works for the Go control plane and the InMemoryChannel broker, but NOT for the Kafka Broker Java data plane (kafka-broker-receiver / kafka-broker-dispatcher).

The Java data plane does NOT: - Mount the ConfigMap - Read config-observability or config-tracing - Have any OTEL env vars by default - Include an OTEL Java agent in the image

This is a known issue.

What DOES work: OTEL Java Agent injection

Inject the OpenTelemetry Java Agent via an init container that downloads the jar, then enable it via JAVA_TOOL_OPTIONS.

Patch the deployment (receiver example):

kubectl patch deploy kafka-broker-receiver -n knative-eventing --type=strategic -p '
spec:
  template:
    spec:
      initContainers:
      - name: otel-agent
        image: curlimages/curl
        command: ["sh", "-c", "curl -L -o /otel/opentelemetry-javaagent.jar https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/v2.6.0/opentelemetry-javaagent.jar"]
        volumeMounts:
        - name: otel-agent
          mountPath: /otel
      volumes:
      - name: otel-agent
        emptyDir: {}
      containers:
      - name: kafka-broker-receiver
        env:
        - name: JAVA_TOOL_OPTIONS
          value: "-javaagent:/otel/opentelemetry-javaagent.jar"
        - name: OTEL_SERVICE_NAME
          value: "kafka-broker-receiver"
        - name: OTEL_TRACES_EXPORTER
          value: "otlp"
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: "http://otel-collector.observability.svc.cluster.local:4317"
        - name: OTEL_EXPORTER_OTLP_PROTOCOL
          value: "grpc"
        - name: OTEL_METRICS_EXPORTER
          value: "none"
        - name: OTEL_LOGS_EXPORTER
          value: "none"
        volumeMounts:
        - name: otel-agent
          mountPath: /otel'

Same pattern for kafka-broker-dispatcher (StatefulSet).

Note: The ghcr.io/open-telemetry/opentelemetry-java-instrumentation/agent:2.6.0 image does NOT exist. Use curlimages/curl to download the jar from GitHub releases.

Why this is needed

The OTEL Java agent auto-instruments: - All HTTP server/client calls (Vert.x used by the Kafka Broker) - Kafka producer/consumer operations - CloudEvent attribute extraction as span attributes\n It propagates W3C traceparent headers automatically on outbound HTTP calls (trigger dispatch).

---

Demo Backend (Python/FastAPI)

The demo backend (demo/backend/) is instrumented with: - opentelemetry-sdk — trace provider + OTLP exporter - opentelemetry-instrumentation-httpx — auto-propagates traceparent on httpx calls - opentelemetry-instrumentation-fastapi — creates spans for inbound requests

This means: 1. When the UI sends an event via POST /api/send, a root span is created 2. The publish to the broker includes traceparent header automatically 3. The broker (with OTEL agent) creates child spans 4. The trigger dispatch back to the backend (POST /events) is traced 5. Everything shares one trace ID

Environment variables

Set on the backend deployment:

env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://otel-collector.observability.svc.cluster.local:4317"
  - name: OTEL_SERVICE_NAME
    value: "demo-backend"

---

Camel-K Integrations

Camel-K has native OTEL support via the telemetry trait:

spec:
  traits:
    telemetry:
      configuration:
        enabled: true
        endpoint: http://otel-collector.observability.svc.cluster.local:4317
        serviceName: broker-to-asb

This is already configured on both integrations in k8s/integrations/.

---

Deployment Steps

# 1. Deploy observability stack
kubectl apply -f k8s/observability/

# 2. Patch kafka-broker-receiver with OTEL agent (see command above)
# 3. Patch kafka-broker-dispatcher with OTEL agent (same pattern)

# 4. Rebuild and deploy demo backend (includes OTEL deps now)
# 5. Re-apply integrations if telemetry trait was added
kubectl apply -f k8s/integrations/

# 6. Access Jaeger UI
kubectl port-forward -n observability svc/jaeger 16686:16686
# Open http://localhost:16686

---

Expected Trace in Jaeger

[demo-backend] POST /api/send
  └── [demo-backend] publish com.example.demo
       └── [kafka-broker-receiver] POST /default/default
            └── [kafka-broker-dispatcher] trigger.dispatch
                 └── [demo-backend] POST /events

All under one trace ID. Each span shows timing, enabling full latency analysis per hop.

---

Troubleshooting

Symptom	Cause	Fix
No traces in Jaeger	Collector can't reach Jaeger	Check network policy (intra-namespace!)
Collector receives nothing	Source not instrumented	Check OTEL agent env vars, restart pods
Traces arrive but disconnected	No traceparent propagation	Ensure httpx instrumented (Python) or OTEL agent (Java)
Image pull error on OTEL agent	Wrong image path	Use curlimages/curl + download from GitHub releases
jaeger:2 image not found	Tag doesn't exist	Use jaeger:latest
Kafka broker ignores ConfigMap	Known issue #4190	Use OTEL Java agent (not ConfigMap)

Production Considerations

• Replace Jaeger all-in-one with persistent storage (Elasticsearch, ClickHouse)
• Set sampling rate < 1.0 for high-throughput
• Add metrics pipeline in OTEL Collector (Prometheus exporter)
• Consider building custom kafka-broker images with OTEL agent baked in (avoids curl download at startup)
• Pin OTEL agent version to avoid breaking changes