Solution Document for Data Streams Monitoring (DSM) Enablement on AKS with RabbitMQ

(Datadog APM via Terraform)

Purpose of the Document

Enable DSM visibility for queue-based microservices running on AKS using RabbitMQ as the message broker and Datadog APM with auto-instrumentation.

What DSM Solves

DSM addresses invisibility in distributed queue pipelines:

  • Throughput
  • Queue latency
  • Processing latency
  • Consumer lag
  • Retries & failures
  • Full producer → queue → consumer correlation

Architecture Scope

In Scope:

  • AKS Cluster
  • RabbitMQ Broker
  • Multi-language services (.NET, Java, Python, Node, Go)
  • Datadog Terraform deployment model
  • Admission Controller for tracer injection

RabbitMQ is not instrumented; spans are captured from producers & consumers.

Deployment Model

DSM involves 3 key layers:

Layer Function
Datadog Agent Collects APM & DSM spans
Admission Controller Auto-injects tracers
Application Runtime Emits queue spans

Why Admission Controller?

  • Zero code changes
  • No rebuild
  • No library updates
  • Multi-language support
  • Fast enablement

Deployment — Datadog Agent via Terraform

Final Values File (YAML)

YAML
datadog:
  site: datadoghq.eu
  apiKeyExistingSecret: datadog-secret

  logs:
    enabled: true
    containerCollectAll: true

  apm:
    enabled: true
    instrumentation:
      enabled: true

  env:
    - name: DD_DATA_STREAMS_ENABLED
      value: "true"
    - name: DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED
      value: "true"

  serviceMonitoring:
    enabled: true

  networkMonitoring:
    enabled: true

  processAgent:
    enabled: true

  containerRuntime:
    enabled: true

  systemProbe:
    enabled: true

  sysctlEnabled: true

  seccompProfileEnabled: false
  appArmorProfileEnabled: false

  admissionController:
    enabled: true
    mutateUnlabelled: true
    failurePolicy: Ignore

  clusterAgent:
    enabled: true

Apply via Terraform

Bash
terraform apply

Purpose: Installs/updates Datadog Agent + Admission Controller + Runtime instrumentation.

Runtime Enablement — Application Layer

Terraform cannot instrument workloads directly; Kubernetes annotations enable DSM:

Bash
kubectl annotate deployment virtual-customer admission.datadoghq.com/enabled=true --overwrite
kubectl annotate deployment order-service admission.datadoghq.com/enabled=true --overwrite
kubectl annotate deployment makeline-service admission.datadoghq.com/enabled=true --overwrite
kubectl annotate deployment virtual-worker admission.datadoghq.com/enabled=true --overwrite

Enable runtime DSM flags:

Bash
kubectl set env deployment/virtual-customer \
DD_DATA_STREAMS_ENABLED=true \
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

kubectl set env deployment/order-service \
DD_DATA_STREAMS_ENABLED=true \
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

kubectl set env deployment/makeline-service \
DD_DATA_STREAMS_ENABLED=true \
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

kubectl set env deployment/virtual-worker \
DD_DATA_STREAMS_ENABLED=true \
DD_TRACE_REMOVE_INTEGRATION_SERVICE_NAMES_ENABLED=true

Language Support Notes

Microservice Role DSM Support
virtual-customer Producer ✔
order-service Producer ✔
makeline-service Consumer ✔
virtual-worker Consumer ✔
rabbitmq Broker N/A

Maturity: - Fully supported → .NET, Java, Python, Node - Partial → Go

Validation Workflow

DSM validation requires strict sequence.

Step 1 — Validate APM

APM → Services expects:

  • virtual-customer
  • order-service
  • makeline-service
  • virtual-worker

If APM is empty → DSM topology cannot build.

Step 2 — Validate DSM Topology

APM → Data Streams Monitoring → Explore

Topology:

Text Only
virtual-customer → rabbitmq → virtual-worker

Metrics expected:

  • Throughput
  • Queue time
  • Processing time
  • Consumer lag
  • Retries
  • Latency

Troubleshooting

Bash
kubectl logs <pod> | grep -i datadog

Expected:

Text Only
Datadog Tracer initialized
Data Streams: Enabled

Operational Notes

DSM requires full path:

Text Only
Producer → Queue → Consumer

Missing either breaks topology reconstruction.

Contact

For more information, contact Airowire Solutions: