Skip to main content
Mapagam

Configuring Observability

1. Understanding Telemetry API

Field Type Description Default
metrics object Configuration for metrics collection and export {}
logs object Configuration for log collection and export {}
traces object Configuration for trace collection and export {}

Example: Basic Telemetry Configuration

telemetry:
  metrics:
    prometheus:
      enabled: true
      port: 9090
  logs:
    level: info
  traces:
    jaeger:
      enabled: true
      endpoint: http://jaeger-collector:14268/api/traces
Note: The Telemetry API allows for flexible configuration of observability tools like Prometheus, Jaeger, and Fluentd.

2. Configuring Telemetry Providers

Provider Supported Metrics Configuration Notes
Prometheus istio_requests_total, istio_tcp_connections_opened_total 
{
  "prometheus": {
    "port": "9090",
    "service": "prometheus-service"
  }
}
 Default metrics collector for Istio
Jaeger istio_request_duration_seconds, istio_request_bytes 
{
  "jaeger": {
    "endpoint": "http://jaeger-collector:14268/api/traces"
  }
}
 Tracing backend for distributed tracing
Grafana istio_response_time_seconds, istio_errors_total 
{
  "grafana": {
    "url": "http://grafana:3000"
  }
}
 Visualization dashboard for metrics

Example: Configuring Prometheus

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  prometheus:
    port: "9090"
    service: prometheus-service

Example: Configuring Jaeger

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  jaeger:
    endpoint: http://jaeger-collector:14268/api/traces

3. Setting Up Metrics Collection

Example: Prometheus Configuration for Service Mesh

scrape_configs:
  - job_name: 'istio-mesh'
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      target_label: __address__
      regex: (.+)
      separator: ;
Configuration Option Description Default Value Notes
job_name Name of the scrape job None Must be unique within Prometheus config
kubernetes_sd_configs Kubernetes service discovery configuration None Required for Kubernetes environments
relabel_configs Relabeling rules to modify labels before scraping None Used to filter and transform metrics
Note: Ensure that the Prometheus instance has proper RBAC permissions to access Kubernetes API for pod discovery.
Warning: Misconfigured relabeling can cause metric loss or incorrect labeling.

Key Steps for Metrics Collection Setup

  • Enable metrics scraping in the service mesh control plane
  • Configure Prometheus to discover pods with appropriate annotations
  • Set up relabeling rules for filtering and enriching metrics
  • Verify scraping targets in Prometheus UI

4. Configuring Tracing Backend

Property Values Description
tracing.backend jaeger, zipkin, otel Specifies the tracing backend to use for collecting and visualizing trace data.
tracing.endpoint http://jaeger-collector:14268/api/traces Endpoint URL where trace data is sent to the backend.
tracing.sampleRate 0.1, 1.0 Percentage of traces to sample (e.g., 0.1 = 10%).

Example: Jaeger Tracing Configuration

tracing:
  backend: jaeger
  endpoint: http://jaeger-collector:14268/api/traces
  sampleRate: 0.1
Note: Ensure that the tracing backend is accessible from the service mesh control plane and data plane components.

5. Setting Up Access Logging

Configuration Option Description Default Value Notes
accessLog.format Format of the access log entries json Supports json, text, and protobuf
accessLog.path Path to the access log file /dev/stdout Can be a file path or /dev/stdout for console output
accessLog.filter Filter to apply before logging none Can filter by status code, response time, etc.

Example: JSON Access Log Configuration

accessLog:
  format: json
  path: /var/log/access.log
  filter:
    statusCode: 200
Note: Ensure the logging path has appropriate permissions for the service mesh to write logs.
Warning: Enabling detailed access logging can significantly impact performance and disk usage.

6. Managing Telemetry Filters

Filter Description Configuration Notes
metrics Collects and exports metrics from the service mesh 
{
  "metrics": {
    "providers": [
      {
        "name": "prometheus",
        "port": 12345
      }
    ]
  }
}
 Default provider is Prometheus
tracing Enables distributed tracing for services 
{
  "tracing": {
    "provider": "jaeger",
    "port": 12346
  }
}
 Jaeger is the default tracing backend
logs Collects logs from services and proxies 
{
  "logs": {
    "format": "json",
    "output": "/var/log/telemetry"
  }
}
 Logs can be exported to external systems

Example: Enabling Metrics Collection

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
spec:
  metrics:
  - providers:
    - name: prometheus

Example: Configuring Tracing

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
spec:
  tracing:
    provider: jaeger
    port: 12346
Note: Telemetry filters can be applied at the mesh, namespace, or workload level.
Warning: Ensure that telemetry providers are configured securely to prevent data leaks.

7. Configuring Custom Telemetry

Property Description Example Notes
telemetry.custom.metrics Custom metrics to be collected and exported custom.metrics: [request_count, error_rate] Supports Prometheus format
telemetry.custom.traces Custom trace configurations custom.traces: [span_name, duration] Can include custom span attributes
telemetry.custom.logs Custom log formats and fields custom.logs: [level, service_name] Supports structured logging

Example: Custom Metrics Configuration

telemetry:
  custom:
    metrics:
      - name: request_count
        description: Total number of requests
        type: counter
      - name: error_rate
        description: Error rate per second
        type: gauge

Example: Custom Trace Configuration

telemetry:
  custom:
    traces:
      - span_name: api_call
        attributes:
          - name: service_name
            value: "user-service"
          - name: endpoint
            value: "/api/users"
Note: Custom telemetry configurations must align with the service mesh's observability backend (e.g., Prometheus, Jaeger).

8. Optimizing Telemetry Overhead

Aspect Description Optimization Strategy
Sampling Rate Percentage of traces to sample for analysis Reduce sampling rate in production to 1-5% to minimize overhead while maintaining observability
Trace Exporter Component responsible for sending trace data to backend systems Use batching and compression to reduce network overhead; configure exporters to send data in bulk
Metrics Collection Frequency and scope of metrics collection Implement adaptive sampling or aggregation to reduce cardinality and frequency in high-volume services
Log Level Verbosity of logs collected Set log levels to INFO or WARN in production to reduce noise and processing overhead
Resource Limits Memory and CPU limits for telemetry agents Set strict resource limits to prevent telemetry components from consuming excessive system resources

Example: Adjusting Sampling Rate in Istio

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: default
spec:
  tracing:
    sampling: 5
Note: Sampling rate adjustments should be made carefully to ensure that critical trace data is not lost during high-traffic periods.

9. Integrating with APM Tools

APM Tool Integration Method Supported Metrics Notes
Datadog datadog-agent with istio integration istio_requests_total, istio_request_duration_seconds Requires datadog-agent to be deployed in the same namespace as Istio
New Relic Uses newrelic-istio-adapter request_count, response_time Supports both Prometheus and OpenTelemetry
Prometheus + Grafana Uses istio-metrics and prometheus-adapter istio_requests_total, istio_request_duration_seconds Highly customizable dashboards with Grafana
Elastic APM Uses elastic-apm-agent with istio integration span_count, error_rate Requires elastic-apm-agent to be injected into application pods

Example: Datadog Integration

apiVersion: v1
kind: ConfigMap
metadata:
  name: datadog-config
data:
  datadog.yaml: |
    istio:
      enabled: true
      service:
        name: datadog-agent
        port: 8126

Example: New Relic Integration

apiVersion: v1
kind: ConfigMap
metadata:
  name: newrelic-config
data:
  newrelic.yaml: |
    istio:
      enabled: true
      service:
        name: newrelic-istio-adapter
        port: 8080
Note: Ensure that the APM tool's agent is configured to collect metrics from the service mesh components.

10. Managing Telemetry Retention

Property Values Description
retention.days 30 Number of days to retain telemetry data
retention.size 10GB Maximum size of telemetry storage before rotation
retention.policy delete Policy for handling expired data: delete, archive

Example: Configuring Retention Policy

telemetry:
  retention:
    days: 30
    size: 10GB
    policy: delete
Note: Retention settings apply to all telemetry types (metrics, logs, traces) unless overridden per type.

11. Implementing Telemetry Sampling

Sampling Type Description Configuration Notes
trace sampling rate Percentage of traces to sample for telemetry 0.1 Default is 10% to reduce overhead
span sampling rate Percentage of spans to sample per trace 0.5 Can be adjusted for performance tuning
metrics sampling rate Percentage of metrics to sample for aggregation 1.0 Full metrics collection for accurate analysis

Example: Configuring Sampling in Istio

apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
  name: mesh-default
spec:
  tracing:
    sampling: 10
  metrics:
    providers:
    - name: prometheus
Note: Sampling rates can be adjusted dynamically without restarting services.

12. Configuring Telemetry Export

Property Description Values Notes
exporter Specifies the telemetry exporter to use otlp, zipkin, jaeger Defaults to otlp
endpoint Endpoint URL for the telemetry collector http://otel-collector:4317 Required for otlp exporter
protocol Protocol used for communication with collector grpc, http/protobuf Defaults to grpc

Example: OTLP Exporter Configuration

telemetry:
  exporters:
    otlp:
      endpoint: http://otel-collector:4317
      protocol: grpc

Example: Zipkin Exporter Configuration

telemetry:
  exporters:
    zipkin:
      endpoint: http://zipkin:9411/api/v2/spans

Was this article helpful?

Related Cheat Sheets