Skip to main content
Mapagam

Monitoring Production Stack

1. Understanding Stack Monitoring

Component Description Functionality
Elasticsearch Search and analytics engine Stores and indexes logs and metrics data for real-time search and analysis
Logstash Server-side data processing pipeline Collects, enriches, and transforms logs before sending to Elasticsearch
Kibana Analytics and visualization platform Provides dashboards, visualizations, and UI for monitoring logs and metrics
Beats Lightweight shipper tools Collect and send logs, metrics, and other data to Logstash or Elasticsearch

Example: Basic Stack Monitoring Setup

# Install and start Elasticsearch
sudo systemctl start elasticsearch

# Install and start Logstash
sudo systemctl start logstash

# Install and start Kibana
sudo systemctl start kibana

# Install Filebeat to ship logs
sudo apt install filebeat
Note: Stack monitoring requires proper configuration of each component to ensure data flow and visualization capabilities.

Key Monitoring Concepts

  • Log Aggregation: Centralized collection of logs from multiple sources
  • Real-time Analysis: Instant processing and querying of log data
  • Alerting: Automated notifications based on log or metric thresholds
  • Dashboards: Customizable visual representations of system health

2. Using Metricbeat for Monitoring

Field Description Example
metricbeat.modules Defines which modules to enable for collecting metrics [{ "name": "system", "enabled": true }]
metricbeat.config.modules Path to module configuration files modules.d/*.yml
output.elasticsearch Configures where to send collected metrics { "hosts": ["localhost:9200"] }

Example: Basic Metricbeat Configuration

metricbeat.modules:
  - name: system
    enabled: true

output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "metricbeat-%{[version]}-%{+yyyy.MM.dd}"
Note: Ensure Elasticsearch is running before starting Metricbeat.

Example: Enabling System Module

metricbeat.modules:
  - name: system
    enabled: true
    metricsets:
      - cpu
      - memory
      - filesystem
Metricset Description Available in
cpu Collects CPU usage statistics system
memory Collects memory usage statistics system
filesystem Collects filesystem usage statistics system

Example: Collecting Disk I/O Metrics

metricbeat.modules:
  - name: system
    enabled: true
    metricsets:
      - diskio
Warning: High-frequency metric collection can impact system performance.

Key Features of Metricbeat

  • Lightweight and efficient
  • Supports various system metrics
  • Can integrate with Elasticsearch and Kibana

3. Configuring Monitoring Cluster

Component Configuration Description
Elasticsearch cluster.name: elk-monitoring Name of the monitoring cluster
Elasticsearch node.name: monitoring-node-1 Unique name for the node
Elasticsearch network.host: 0.0.0.0 Allow connections from any IP address
Elasticsearch http.port: 9200 HTTP port for API access
Elasticsearch xpack.security.enabled: true Enable security features
Elasticsearch xpack.monitoring.enabled: true Enable monitoring capabilities
Elasticsearch xpack.monitoring.collection.enabled: true Enable collection of monitoring data
Elasticsearch xpack.monitoring.collection.interval: 10s Interval between data collection

Example: Elasticsearch Configuration File

cluster.name: elk-monitoring
node.name: monitoring-node-1
network.host: 0.0.0.0
http.port: 9200
xpack.security.enabled: true
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true
xpack.monitoring.collection.interval: 10s
Note: Ensure that the monitoring cluster has sufficient resources to handle data collection and storage without impacting performance of the main production cluster.

4. Viewing Cluster Metrics

Metric Description Collection Method Notes
CPU Usage Percentage of CPU cores utilized by the cluster nodes Node Stats API, os.cpu.percent High CPU usage may indicate performance bottlenecks
Memory Usage Amount of RAM used by Elasticsearch processes Node Stats API, os.mem.used_in_bytes Memory pressure can lead to GC overhead or out-of-memory errors
Disk I/O Read/write operations on disk storage Node Stats API, os.fs.io_stats High I/O wait times may suggest disk performance issues
Heap Usage Java heap memory consumption in Elasticsearch JVM Stats API, jvm.mem.heap_used_in_bytes Heap size should be tuned to avoid frequent GC cycles
Thread Pool Rejection Count Number of tasks rejected due to thread pool exhaustion Node Stats API, thread_pool.*.rejected Rejection count increasing indicates overload or misconfiguration
Cluster Health Status Overall health of the cluster (green/yellow/red) Cluster Health API, cluster_health.status Red status indicates unassigned primary shards

Example: Fetch Cluster Metrics via API

GET /_cluster/stats

Example: Get Node-Level Metrics

GET /_nodes/stats
Note: Use Kibana's Monitoring dashboard for real-time visualization of these metrics.

5. Monitoring Index Performance

Property Description Values Notes
indexing throughput Number of documents indexed per second 1000 Measured in ops/sec
search latency Average time to respond to search queries 50ms Typical range: 10-200ms
index size Total storage used by indices 2.5GB Includes all shards and replicas
shard allocation Number of shards per index 5 Optimal for performance and scalability
refresh interval How often index is refreshed for searchability 30s Default is 1s; can be adjusted for performance

Example: Index Performance Query

GET /_cat/indices?v&h=index,docs.count,store.size,pri.store.size,refresh.time

Example: Shard Allocation Check

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason
Note: Use _cat/indices to monitor index size and document count. Adjust refresh interval based on indexing needs.
Warning: High shard count can lead to performance degradation. Aim for 20-40 shards per index.

6. Tracking Query Performance

Property Description Values Notes
elasticsearch.index.search Search query performance metrics latency, throughput Monitored via search API
elasticsearch.index.query Query execution time avg_time, max_time Used in Kibana Visualizations
elasticsearch.index.requests Total query requests count, rate Tracked via GET /_stats

Example: Query Performance Monitoring

GET /_cluster/stats
{
  "nodes": {
    "os": {
      "load_average": [1.2, 1.5, 1.8]
    }
  }
}

Example: Query Latency Metrics

GET /_stats/search
{
  "indices": {
    "my_index": {
      "search": {
        "query_total": 1200,
        "query_time_in_millis": 45000
      }
    }
  }
}
Note: Use elasticsearch.indices to track query performance over time.
Warning: High query latency can indicate resource contention or inefficient queries.

7. Using APM Integration

Example: Setting up APM Integration in Kibana

curl -X POST "http://localhost:5601/api/apm/integrations" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer <access_token>" \
-d &#x27;{
  "name": "MyApp-APM",
  "service": {
    "name": "myapp",
    "environment": "production"
  },
  "agent": "nodejs",
  "config": {
    "serverUrl": "http://localhost:8200"
  }
}&#x27;
Field Description Values Notes
name Integration name "MyApp-APM" Unique identifier for the integration
service.name Service name "myapp" Used to group APM data
service.environment Environment name "production" Used to filter APM data
agent APM agent type "nodejs", "java", "python" Must match the application language
config.serverUrl APM Server URL "http://localhost:8200" Must point to running APM Server
Note: Ensure the APM agent is configured to send data to the correct APM Server endpoint.
Warning: The serverUrl must be accessible from the application environment.

APM Integration Setup Steps

  • Install APM agent in your application
  • Configure the APM agent to point to the APM Server
  • Create an integration in Kibana using the apm/integrations API
  • Verify data is flowing in Kibana APM dashboard

8. Monitoring JVM Metrics

Metric Description Collection Method Notes
jvm.memory.heap.used Heap memory currently used by the JVM JMX, JFR Useful for tracking memory pressure
jvm.memory.heap.committed Heap memory committed to the OS JMX, JFR Indicates how much memory is reserved
jvm.gc.pause Time spent in garbage collection pauses JMX, JFR High values indicate performance issues
jvm.thread.count Number of live threads in the JVM JMX, JFR Can help identify thread contention
jvm.uptime Time since JVM started JMX, JFR Useful for tracking application stability

Example: Collecting JVM Metrics with JMX

curl -X GET http://localhost:9090/jmx?query=java.lang:type=Memory
Note: Ensure that JMX is enabled in your JVM with the appropriate flags for monitoring.
Warning: Frequent GC pauses can indicate memory leaks or inefficient code.

9. Tracking Disk Usage

Component Default Storage Path Notes
Elasticsearch /var/lib/elasticsearch Contains indices, logs, and configuration files
Logstash /var/lib/logstash Stores pipeline configurations and internal state
Kibana /var/lib/kibana Stores saved objects, plugins, and configuration

Example: Check Disk Usage with df Command

df -h /var/lib/elasticsearch

Example: Monitor Elasticsearch Disk Usage via API

curl -X GET "localhost:9200/_cat/nodes?v&h=ip,heapPercent,diskAvail"
Note: Elasticsearch uses a heap.size setting to control memory usage, which indirectly affects disk space due to the use of file system for index storage and caching.

Key Disk Usage Metrics

  • Disk Available - Free space on the volume hosting Elasticsearch indices
  • Heap Percentage - Memory usage of Elasticsearch nodes
  • Index Size - Total size of all indices in the cluster

10. Monitoring Network Traffic

Component Description Usage
Logstash Collects and processes network logs from various sources. Configured with input plugins to capture traffic data.
Elasticsearch Stores and indexes network traffic logs for search and analysis. Used to query logs by IP, port, or protocol.
Kibana Visualizes network traffic through dashboards and graphs. Provides real-time monitoring of network activity.

Example: Logstash Configuration for Network Logs

{
  "input": {
    "tcp": {
      "port": 5000,
      "codec": "json"
    }
  },
  "filter": {
    "grok": {
      "match": {
        "message": "%{IP:client_ip} %{WORD:method} %{URIPATH:uri} %{NUMBER:bytes}"
      }
    }
  },
  "output": {
    "elasticsearch": {
      "hosts": ["localhost:9200"],
      "index": "network-logs-%{+YYYY.MM.dd}"
    }
  }
}
Note: Ensure that Logstash is configured to handle high-volume traffic logs efficiently.
Warning: Network traffic logs can be large; consider using gzip compression to reduce bandwidth usage.

Key Points for Network Traffic Monitoring

  • Use tcp or udp inputs in Logstash to capture traffic logs.
  • Filter logs using grok patterns to extract meaningful data.
  • Index logs in Elasticsearch with a time-based index pattern for efficient querying.
  • Visualize traffic data using Kibana dashboards with real-time updates.

11. Using Logs App

Example: Viewing Logs in Kibana

{
  "index": "logs-*",
  "query": {
    "match": {
      "service.name": "api-server"
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc"
      }
    }
  ]
}
Field Description Example Value
index Log index pattern to query logs-*
query Filter to apply to logs {"match": {"service.name": "api-server"}}
sort Sort order for results [{"@timestamp": {"order": "desc"}}]
Note: The Logs app in Kibana provides a real-time view of your logs with filtering and sorting capabilities.

Example: Creating a Log Stream

GET /_cat/indices?v

Logs App vs. Logstash

  • Logs App: Real-time log viewing and filtering
  • Logstash: Log ingestion, transformation, and routing

12. Setting Up Uptime Monitoring

Example: Basic Uptime Monitoring Setup

# elasticsearch.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: [ "http://localhost:9200" ]

# kibana.yml
xpack.monitoring.ui.container.elasticsearch.hosts: [ "http://localhost:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true
Configuration Description Default Value Notes
xpack.monitoring.enabled Enables monitoring in Elasticsearch false Required for monitoring features
xpack.monitoring.elasticsearch.hosts Array of Elasticsearch hosts to monitor [] Must include at least one host
xpack.monitoring.ui.container.elasticsearch.hosts Hosts for Kibana UI monitoring [] Used in Kibana's Monitoring UI
xpack.monitoring.ui.container.elasticsearch.enabled Enables container monitoring in Kibana false Must be enabled for UI container view
Note: Ensure that the Elasticsearch hosts specified in xpack.monitoring.elasticsearch.hosts are accessible from the Elasticsearch node where monitoring is enabled.
Warning: Enabling monitoring can impact performance, especially in high-volume environments. Monitor resource usage after enabling.

Was this article helpful?

Related Cheat Sheets