Skip to main content
Mapagam

Implementing Observability Features

1. Understanding Observability Pillars

Pillar Description Components Tools in ELK
Logs Records events and activities in a system for analysis and troubleshooting. Structured and unstructured text data Logstash, Beats, Elasticsearch
Metrics Quantitative measurements of system performance and behavior over time. Time-series data, counters, gauges Metricsbeat, Elasticsearch, Kibana
Traces Tracks the path of requests through distributed systems to identify bottlenecks. Distributed request tracking, latency data APM Server, Elasticsearch, Kibana

Logs vs Metrics vs Traces

  • Logs: Best for detailed event information, debugging, and audit trails.
  • Metrics: Ideal for real-time monitoring, performance dashboards, and alerting.
  • Traces: Essential for understanding request flow in microservices and distributed systems.

ELK Stack Observability Features

  • Log Management: Collect, parse, and analyze logs using Beats and Logstash.
  • Metrics Collection: Gather system and application metrics with Metricsbeat.
  • Distributed Tracing: Use APM Server to trace requests across services.
  • Visualization & Alerting: Create dashboards in Kibana and set up alerts for anomalies.

2. Using Logs App

Feature Description Usage
Log data ingestion Collects and processes log data from various sources Configure Logstash or Beats to send logs to Elasticsearch
Log visualization Provides real-time dashboards for log analysis Use Kibana's Logs app to create visualizations
Log filtering Allows filtering logs by fields, time range, and keywords Apply filters in the Logs app UI or via query DSL

Example: Basic Log Filtering

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "level": "ERROR"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2025-01-01T00:00:00Z"
            }
          }
        }
      ]
    }
  }
}
Note: The Logs app in Kibana provides a powerful interface for exploring and analyzing log data. It supports real-time log streaming, filtering, and visualization.

3. Using Metrics App

Example: Setting up Metrics App in Kibana

// Navigate to Kibana > Metrics App
// Create a new dashboard with metric queries
{
  "query": "avg(cpu_usage)",
  "timeRange": {
    "from": "now-1h",
    "to": "now"
  },
  "groupBy": ["host"]
}
Feature Description Availability
Metrics App Real-time monitoring and visualization of system metrics NEW in Kibana 8.10
Dashboard Creation Create custom dashboards with metric visualizations Stable
Metric Queries Define queries to aggregate and filter metrics data Stable
Note: The Metrics App leverages Elasticsearch's aggregation engine for efficient querying of time-series data.
Warning: Ensure proper indexing of metric data in Elasticsearch to maintain performance.

Key Capabilities

  • Real-time metric visualization
  • Custom dashboard creation
  • Aggregation and filtering of time-series data
  • Integration with APM for full-stack observability

4. Using Uptime App

Example: Setting up Uptime Monitoring

uptime:
  name: "API Service"
  url: "https://api.example.com/health"
  interval: 30
  timeout: 10
  alert:
    threshold: 3
    message: "Service is down"
Property Description Default
name Name of the monitor None
url Endpoint to monitor None
interval Check frequency in seconds 30
timeout Timeout in seconds 10
alert.threshold Number of failures to trigger alert 3
Note: Uptime checks are essential for monitoring service availability and response times.

Uptime App vs Other Monitoring Tools

  • Uptime App: Simple, focused on endpoint availability
  • Prometheus: Comprehensive metrics collection and alerting
  • New Relic: Full-stack application performance monitoring

5. Correlating Logs and Traces

Field Description Example
trace.id Unique identifier for a trace across services 1234567890abcdef1234567890abcdef
span.id Unique identifier for a span within a trace abcdef1234567890
log.trace.id Correlation ID used to link logs to traces 1234567890abcdef1234567890abcdef

Example: Log Entry with Trace ID

{
  "timestamp": "2026-01-25T10:00:00Z",
  "level": "INFO",
  "message": "User login successful",
  "trace.id": "1234567890abcdef1234567890abcdef",
  "span.id": "abcdef1234567890"
}

Example: Trace Context in HTTP Headers

GET /api/users/123
X-Trace-ID: 1234567890abcdef1234567890abcdef
X-Span-ID: abcdef1234567890
Note: Ensure consistent trace ID propagation across services to enable end-to-end correlation.

Correlation Best Practices

  • Use standardized trace IDs across all services in the stack
  • Propagate trace context via HTTP headers or message queues
  • Include trace and span IDs in log entries for debugging
  • Use OpenTelemetry or similar standards for interoperability

6. Using Infrastructure App

Example: Basic Infrastructure App Configuration

infrastructure:
  app:
    name: "my-service"
    version: "1.0.0"
    environment: "production"
    monitoring:
      enabled: true
      endpoints:
        - "/metrics"
        - "/health"
Property Description Default Notes
infrastructure.app.name Name of the application undefined Required field
infrastructure.app.version Version of the application undefined Required field
infrastructure.app.environment Environment where app is running (e.g., production, staging) "development" Used for filtering logs and metrics
infrastructure.app.monitoring.enabled Enable or disable monitoring false Set to true to enable metrics and health checks
infrastructure.app.monitoring.endpoints List of endpoints to expose for monitoring [] Supports /metrics and /health by default
Note: The infrastructure app configuration should align with the ELK stack's log and metric collection capabilities to ensure observability.

7. Creating SLOs

Example: Define an SLO for response time

name: "Response time SLO"
description: "95% of requests should complete within 200ms"
objective: 0.95
measurement: "http.response.duration"
threshold: "<= 200ms"
time_window: "24h"
Field Description Values
name Name of the SLO string
description Human-readable description of the SLO string
objective Target percentage (e.g., 0.95 = 95%) number
measurement Metrics to measure (e.g., response time) string
threshold Threshold for the metric (e.g., "<= 200ms") string
time_window Time window over which to calculate SLO (e.g., "24h") string
Note: SLOs are defined in terms of objective, measurement, and threshold. These define what constitutes a "good" event for the service.

Key Concepts

  • SLOs define the desired level of service quality.
  • SLIs are the metrics used to measure SLO compliance.
  • SLIs must be actionable and tied to user experience.

SLO vs SLI vs SLA

  • SLO: A target percentage of service quality (e.g., 99.9% uptime)
  • SLI: A measurable metric (e.g., "95% of requests complete in < 200ms")
  • SLA: A contractual agreement between service provider and consumer

8. Using Observability Rules

Rule Description Configuration Example
trigger Condition that activates the rule threshold, anomaly, change 
{
  "trigger": {
    "threshold": {
      "value": 90,
      "operator": "gt"
    }
  }
}
actions Actions to execute when triggered alert, log, webhook 
{
  "actions": [
    {
      "type": "alert",
      "severity": "high"
    }
  ]
}
conditions Logical conditions to evaluate and, or 
{
  "conditions": [
    {
      "field": "status",
      "operator": "eq",
      "value": "error"
    }
  ]
}

Example: Alert Rule Configuration

{
  "name": "High CPU Usage Alert",
  "trigger": {
    "threshold": {
      "value": 80,
      "operator": "gt"
    }
  },
  "conditions": [
    {
      "field": "cpu.utilization",
      "operator": "gt",
      "value": 80
    }
  ],
  "actions": [
    {
      "type": "alert",
      "severity": "medium"
    }
  ]
}
Note: Rules can be configured to trigger based on metrics, logs, or traces.
Warning: Avoid overly sensitive rules that may cause alert fatigue.

9. Viewing Service Overview

Example: Kibana Service Map

{
  "serviceMap": {
    "serviceName": "user-service",
    "spanCount": 1200,
    "errorRate": "0.5%",
    "latency": "120ms"
  }
}
Property Description Value Type Notes
serviceName Name of the service string Used for filtering and grouping
spanCount Total number of spans in the service integer Indicates service activity level
errorRate Percentage of failed requests string (percentage) Triggers alerts when above threshold
latency Average response time in milliseconds string (duration) Used for performance monitoring
Note: The service overview dashboard in Kibana provides a centralized view of all services, including their health status, performance metrics, and error rates.
Warning: Ensure that the service map is updated regularly to reflect real-time changes in your infrastructure.

Key Metrics for Service Overview

  • Span Count - Total number of spans in the service
  • Error Rate - Percentage of failed requests
  • Latency - Average response time in milliseconds
  • Service Name - Unique identifier for the service

10. Using Log Categorization

Log categorization in the ELK Stack helps organize and classify log entries for easier analysis and monitoring. It allows you to group logs by type, severity, or application component.

Category Description Use Case Example
Error Logs Logs indicating failures or exceptions in applications Debugging application issues {"level": "ERROR", "message": "Database connection failed"}
Info Logs General operational messages that provide insight into system behavior Monitoring application flow {"level": "INFO", "message": "User login successful"}
Warning Logs Messages indicating potential issues that don't stop the application Identifying possible problems before they escalate {"level": "WARN", "message": "High memory usage detected"}

Example: Categorizing Logs with Logstash

{
  "filter": [
    {
      "grok": {
        "match": {
          "message": "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}"
        }
      }
    },
    {
      "mutate": {
        "add_tag": [
          "error",
          "if [loglevel] == 'ERROR'"
        ]
      }
    }
  ]
}
Note: Categorization can be done using Logstash filters or by configuring Kibana's Log Categorization feature.

Key Benefits of Log Categorization

  • Improves search and filtering in Kibana
  • Enables faster troubleshooting by grouping related logs
  • Supports automated alerting based on log categories

11. Implementing Distributed Tracing

Component Description Integration
Jaeger Open-source distributed tracing system Integrates with ELK via Logstash or Beats
Zipkin Open-source tracing system for microservices Can be integrated with ELK using custom Logstash plugins
OpenTelemetry Vendor-neutral observability framework Collects and exports traces to ELK via OTLP exporter

Example: OpenTelemetry Configuration

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:

exporters:
  elasticsearch:
    endpoints: [ "http://localhost:9200" ]
    index: "traces"

service:
  pipelines:
    traces:
      receivers: [ otlp ]
      processors: [ batch ]
      exporters: [ elasticsearch ]
Note: Ensure that the ELK stack has proper index templates for trace data to avoid mapping issues.

Tracing Tools Comparison

  • Jaeger: Excellent for Kubernetes environments, supports multiple protocols
  • Zipkin: Simple to set up, good for monolithic applications
  • OpenTelemetry: Vendor-neutral, extensible, supports multiple backends

Implementation Steps

  1. Deploy OpenTelemetry Collector or Jaeger agent in your services
  2. Configure tracing instrumentation for your application
  3. Set up Logstash or Beats to collect trace data
  4. Index traces in Elasticsearch using proper templates
  5. Visualize trace data in Kibana

12. Monitoring Cloud Services

Service Monitoring Capabilities Integration Method Notes
AWS CloudWatch Metrics, Logs, Alarms CloudWatch Agent or FireLens Native integration with ELK Stack via Logstash or Beats
Azure Monitor Metrics, Logs, Alerts Azure Monitor Agent or Logstash Supports Azure AD and RBAC for secure access
Google Cloud Monitoring Metrics, Logs, Trace Stackdriver Agent or Fluentd Provides detailed service-level metrics and logs
Datadog Metrics, Logs, Traces Agent or API integration Supports ELK Stack through custom integrations and dashboards

Example: Configuring CloudWatch Logs with Logstash

# Install the AWS CloudWatch Logs plugin for Logstash
bin/logstash-plugin install logstash-input-cloudwatch_logs

# Configure Logstash to fetch logs from CloudWatch
input {
  cloudwatch_logs {
    region => "us-east-1"
    log_group_name => "/aws/lambda/my-function"
    start_time => "2024-01-01T00:00:00Z"
  }
}
Note: Ensure proper IAM roles and permissions are configured to allow Logstash or Beats to access cloud service logs.
Warning: CloudWatch Logs can incur costs based on log volume and retention settings.

Was this article helpful?

Related Cheat Sheets