Skip to main content
Mapagam

Working with Metrics

1. Understanding Standard Metrics

Metric Type Description Typical Use Case Sample Value
requests_total Total number of requests handled by the service mesh Monitoring request volume over time 12500
request_duration_seconds Distribution of request processing times Identifying performance bottlenecks 0.05
request_size_bytes Size of incoming requests in bytes Capacity planning and optimization 1024
response_size_bytes Size of outgoing responses in bytes Monitoring data transfer efficiency 2048
errors_total Total number of failed requests Tracking error rates and reliability 42
throughput_gbps Network throughput in gigabits per second Measuring network performance 1.5

Example: Prometheus Query for Request Count

sum(rate(istio_requests_total[5m])) by (destination_service)
Note: These standard metrics are typically exposed via Prometheus endpoints and can be visualized using Grafana dashboards.

2. Configuring Prometheus Integration

Example: Basic Prometheus Configuration

prometheus:
  config:
    scrape_configs:
      - job_name: "istio-mesh"
        static_configs:
          - targets: ["istiod.istio-system.svc.cluster.local:15014"]
      - job_name: "istio-proxy"
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            target_label: __address__
            regex: (.+)
            replacement: $1:15020
Configuration Option Description Default Value Notes
prometheus.config.scrape_configs List of scrape configurations Empty list Required for integration
job_name Name of the scraping job None Must be unique per config
static_configs Static target configurations Empty list Used for fixed endpoints
kubernetes_sd_configs Kubernetes service discovery Disabled Enables dynamic target discovery
relabel_configs Relabeling rules for metrics None Used to filter or transform labels
Note: Ensure that Prometheus has proper RBAC permissions to access Kubernetes API for service discovery.
Warning: Misconfigured relabeling can cause metric loss or incorrect labeling.

Key Configuration Points

  • Define scrape jobs for Istio components
  • Use Kubernetes SD to discover proxy metrics
  • Relabeling ensures correct metric labels
  • Verify Prometheus can reach Istio control plane endpoints

3. Setting Metric Collection Intervals

Interval Type Default Value Description Notes
metrics.collection.interval 15s How often metrics are collected from each service Can be adjusted to reduce load or increase granularity
metrics.scrape.interval 30s How frequently Prometheus scrapes metrics from the service mesh Should align with collection interval for consistency
metrics.export.interval 60s How often metrics are exported to external systems (e.g., Grafana) Higher intervals reduce export overhead but may delay dashboard updates

Example: Configuring Metric Collection Intervals in Istio

apiVersion: v1
kind: ConfigMap
metadata:
  name: istio
data:
  values.metrics.collection.interval: "10s"
  values.metrics.scrape.interval: "20s"
  values.metrics.export.interval: "45s"
Note: Shorter intervals increase resource usage but provide more granular data for real-time monitoring.

4. Implementing Custom Metrics

Example: Defining a Custom Metric in Prometheus

groups:
- name: custom-metrics
  rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status="5xx"}[5m]) > 0.01
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "High error rate detected"
      description: "The error rate for HTTP requests is above 1%"
Metric Type Description Usage
Counter Tracks a monotonically increasing value, typically for counting events or operations. http_requests_total
Gauge Represents a single numerical value that can be modified arbitrarily. http_request_duration_seconds
Histogram Measures the distribution of values in a set of observations. http_request_duration_seconds
Summary Similar to histogram but provides quantiles over a sliding time window. http_request_duration_seconds
Note: Custom metrics should be defined with clear labels to enable filtering and aggregation across service instances.

Example: Adding Custom Labels to Metrics

const client = new PrometheusClient();
const customMetric = new client.Counter({
  name: 'custom_requests_total',
  help: 'Total number of custom requests',
  labelNames: ['endpoint', 'method']
});

// Incrementing the metric with labels
customMetric.labels({ endpoint: '/api/users', method: 'GET' }).inc();

Comparison: Prometheus vs. Grafana

  • Prometheus: Pull-based metric collection engine with a powerful query language (PromQL).
  • Grafana: Visualization and dashboarding tool that can query Prometheus metrics.

5. Configuring Metric Dimensions

Dimension Description Configuration Notes
source Identifies the origin of the metric "source" Used for filtering and aggregation
destination Identifies the target of the metric "destination" Helps in tracing request flow
operation Operation or method being performed "operation" Useful for service-level metrics
response_code HTTP response code or status "response_code" Used for error rate tracking
request_size Size of the request payload "request_size" Helps in identifying large payloads
response_size Size of the response payload "response_size" Useful for bandwidth monitoring

Example: Configuring Dimensions in Istio

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: ALL_METRICS
      tagOverrides:
        source:
          value: source.workload
        destination:
          value: destination.workload
        operation:
          value: request.operation
        response_code:
          value: response.code

6. Understanding Control Plane Metrics

Metric Description Unit Notes
istio_requests_total Total number of requests handled by the control plane count Includes all requests, successful and failed
istio_request_duration_seconds Duration of requests to the control plane in seconds seconds Measures latency of control plane operations
istio_requests_size_bytes Size of requests to the control plane in bytes bytes Used for monitoring request payload sizes
istio_response_size_bytes Size of responses from the control plane in bytes bytes Used for monitoring response payload sizes
istio_control_plane_status Status of the control plane (0 = unhealthy, 1 = healthy) gauge Indicates overall control plane health

Example: Querying Control Plane Metrics

curl -H "Authorization: Bearer $(kubectl get secret -n istio-system | grep istio-admin | awk '{print $1}' | xargs kubectl get secret -n istio-system -o jsonpath='{.data.token}' | base64 --decode)" http://istiod.istio-system.svc:15014/metrics
Note: Control plane metrics are exposed via the istiod service on port 15014.
Warning: Ensure proper authentication and authorization when querying control plane metrics in production environments.

7. Understanding Data Plane Metrics

Metric Type Description Typical Use Case Sample Value
request_count Total number of requests handled by the service mesh Monitoring traffic volume 1250
request_duration_seconds Time taken to process a request in seconds Performance analysis and latency tracking 0.15
response_size_bytes Size of response payload in bytes Bandwidth usage and optimization 2048
error_count Number of failed requests or errors Error rate monitoring and alerting 5

Example: Collecting Data Plane Metrics

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: data-plane-metrics
spec:
  selector:
    matchLabels:
      app: istio-proxy
  endpoints:
  - port: metrics
    path: /stats/prometheus
Note: Data plane metrics are collected from sidecars (e.g., Envoy proxies) and typically include latency, throughput, and error rates.

8. Setting Up Metric Alerts

Example: Prometheus Alert Rule Configuration

groups:
- name: service-mesh-alerts
  rules:
  - alert: HighErrorRate
    expr: rate(istio_requests_total{destination_service=~".*",status!~"2.*"}[5m]) > 0.05
    for: 2m
    labels:
      severity: critical
    annotations:
      summary: "High error rate detected"
      description: "The service {{ $labels.destination_service }} is experiencing {{ $value }} errors per second."
Alert Rule Field Description Example Value
alert Name of the alert rule HighErrorRate
expr Prometheus query expression to evaluate rate(istio_requests_total{destination_service=~".*",status!~"2.*"}[5m]) > 0.05
for Duration to wait before firing the alert 2m
labels Additional metadata for the alert { severity: critical }
annotations Human-readable descriptions and summaries { summary: "High error rate detected", description: "..." }
Note: Alert rules must be defined in Prometheus configuration files or via the Prometheus API. The for duration helps prevent flapping alerts by ensuring the condition persists for a specified time.

Workflow: Configuring Metric Alerts in Istio

  1. Define alert rules in Prometheus configuration files or using the Prometheus API
  2. Ensure the service mesh's metrics are exposed via Prometheus (e.g., istio_requests_total)
  3. Validate the Prometheus query expression using the Prometheus console or web UI
  4. Configure alertmanager to receive and route alerts (e.g., via Slack, email)
  5. Deploy updated configurations to the service mesh and monitoring stack

Key Metrics for Alerting

  • istio_requests_total: Total number of requests, useful for detecting traffic anomalies or errors
  • istio_request_duration_seconds: Request latency, useful for identifying performance degradation
  • istio_requests_size_bytes: Request size, useful for detecting unusual traffic patterns
  • istio_response_size_bytes: Response size, useful for monitoring data flow

9. Configuring Metric Aggregation

Property Description Default Value Notes
aggregation.window Time window for metric aggregation 5m Supported units: s, m, h
aggregation.type Aggregation method for metrics sum Options: sum, avg, max, min
aggregation.metrics List of metrics to aggregate ["requests", "latency"] Can include custom metric names

Example: Aggregation Configuration

metrics:
  aggregation:
    window: 10m
    type: avg
    metrics:
      - requests
      - latency

10. Troubleshooting with Metrics

Example: Identifying Performance Bottlenecks

apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
spec:
  metrics:
  - providers:
    - name: prometheus
    overrides:
    - match:
        metric: ALL_METRICS
      tagOverrides:
        request_duration:
          operation: "request_duration_ms"
Metric Description Use Case Alert Threshold
request_duration Time taken to process a request Identify slow services or endpoints > 500ms
request_count Total number of requests per second Detect traffic spikes or drops < 100 req/s (anomaly)
error_rate Percentage of failed requests Monitor service reliability > 5% (over 5 min)
Note: Use Grafana dashboards to visualize these metrics over time for effective troubleshooting.
Warning: Ensure that metrics collection does not introduce significant latency into your service mesh.

Common Troubleshooting Scenarios

  • High latency: Check request_duration and error_rate for outliers.
  • Service downtime: Monitor error_rate and request_count for sudden drops.
  • Traffic anomalies: Use request_count to detect unusual traffic patterns.

11. Implementing Metric Dashboards

Example: Grafana Dashboard Setup

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-config
data:
  dashboard.json: |
    {
      "dashboard": {
        "title": "Service Mesh Metrics",
        "panels": [
          {
            "title": "Request Rate",
            "type": "graph",
            "targets": [
              {
                "expr": "rate(istio_requests_total[5m])"
              }
            ]
          }
        ]
      }
    }
Dashboard Component Description Configuration
Panel Visual representation of metrics type: graph, targets
Query Expression to fetch metrics rate(istio_requests_total[5m])
Data Source Backend system providing metrics Grafana with Prometheus backend
Note: Ensure the service mesh exposes metrics in Prometheus format for integration with Grafana.
Warning: Dashboard configurations may require specific permissions to access metrics endpoints.

12. Managing Metric Cardinality

Note: High metric cardinality can lead to performance degradation and increased storage requirements in monitoring systems.
Metric Type Description Cardinality Impact Best Practices
Request Count Counts HTTP/GRPC requests per service Low to Medium Use labels like method, status to group
Latency Measures response times Medium Avoid high-resolution histograms; use buckets wisely
Error Rate Tracks failed requests Medium Aggregate errors by service and endpoint
Traffic Volume Tracks bytes transferred High Sample or aggregate traffic data to reduce cardinality

Example: Reducing High Cardinality Metrics

metric:
  name: "request_count"
  labels:
    - method
    - status
    - service
  # Avoid including user_id or other high-cardinality fields

Strategies to Reduce Metric Cardinality

  • Label Filtering: Only include low-cardinality labels in metrics
  • Aggregation: Group similar metrics into buckets or summaries
  • Sampling: Use sampling techniques for high-volume metrics
  • Time-based Buckets: Reduce resolution over time to save resources

Cardinality vs. Performance Trade-offs

  • High Cardinality: More granular data, but higher memory and CPU usage
  • Low Cardinality: Less granular, but more efficient for storage and querying

Was this article helpful?

Related Cheat Sheets