Debugging and Troubleshooting

1. Checking Elasticsearch Logs

Log Type	Location	Description
Elasticsearch	`/var/log/elasticsearch/`	Default log directory for Elasticsearch service logs
Application	`/var/log/elasticsearch/<cluster-name>.log`	Primary application log file with detailed runtime information
GC Logs	`/var/log/elasticsearch/<cluster-name>-gc.log`	Garbage collection logs for JVM performance monitoring

Example: Viewing Elasticsearch Logs

# View recent logs using tail
tail -f /var/log/elasticsearch/my-cluster.log

# Check for errors in the last hour
grep -i "error" /var/log/elasticsearch/my-cluster.log | tail -n 20

# Monitor GC activity
tail -f /var/log/elasticsearch/my-cluster-gc.log | grep "GC"

Note: Ensure proper permissions to read log files. Logs may require sudo access on some systems.

Warning: Log files can grow very large. Use tools like logrotate to manage disk space.

2. Checking Logstash Logs

Log Level	Description	Typical Use Case
ERROR	Logs errors that prevent a feature from functioning.	Identifying failed pipeline processing or configuration issues.
WARN	Logs events that might be problematic but don't stop processing.	Monitoring potential issues like dropped events or deprecated settings.
INFO	General operational messages indicating normal flow.	Tracking successful pipeline execution or plugin loading.
DEBUG	Detailed information for diagnosing problems.	Investigating specific event transformations or plugin behavior.

Example: Logstash Configuration for Verbose Logging

{
  "pipeline": {
    "workers": 2,
    "batch_size": 125,
    "log_level": "debug"
  }
}

Note: Logstash logs are typically found in /var/log/logstash/ on Linux systems.

Warning: Enabling debug logging can significantly impact performance and disk space usage.

3. Checking Kibana Logs

Log Type	Location	Description	Notes
Kibana Server Logs	`/var/log/kibana/kibana.log`	Primary log file for Kibana server activity	Default location; may vary by installation
Kibana Server Logs (Verbose)	`/var/log/kibana/kibana.log`	Verbose logging with detailed error traces	Enable via `kibana.yml` setting `logging.verbose: true`
Elasticsearch Logs	`/var/log/elasticsearch/`	Logs from the Elasticsearch backend	Check `elasticsearch.log` for cluster health and indexing errors

Example: View Kibana Logs

tail -f /var/log/kibana/kibana.log

Example: Filter Kibana Logs for Errors

grep -i "error" /var/log/kibana/kibana.log

Note: Ensure proper file permissions to read Kibana logs. On some systems, you may need sudo access.

Warning: Kibana logs can grow very large. Use tools like logrotate to manage log size.

4. Using Verbose Logging

Log Level	Description	Use Case
TRACE	Most detailed logging level, includes all internal operations.	Debugging complex issues or performance bottlenecks in development.
DEBUG	Provides detailed information for diagnosing problems.	Identifying root causes during application debugging or troubleshooting.
INFO	General operational messages indicating normal system behavior.	Monitoring application flow and confirming expected operations.
WARN	Warning messages indicating potential issues that do not stop execution.	Identifying non-critical problems or deprecated practices.
ERROR	Error messages indicating failures in operations that prevent expected behavior.	Tracking failures or exceptions that require attention.
FATAL	Severe errors that cause the application to terminate.	Monitoring critical failures or system crashes.

Example: Configuring Verbose Logging in Logstash

# Set log level to DEBUG in Logstash configuration
pipeline {
  input {
    beats {
      port => 5044
    }
  }
  filter {
    grok {
      match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{MESSAGE:message}" }
    }
  }
  output {
    elasticsearch {
      hosts => [ "localhost:9200" ]
      index => "logstash-%{+YYYY.MM.dd}"
    }
  }
}

Example: Enabling Verbose Logging in Elasticsearch

# elasticsearch.yml
logger.level: DEBUG

# Enable detailed logging for specific components
logger.indexing: TRACE
logger.search: DEBUG

Note: Verbose logging can significantly impact performance and disk space usage. It is recommended to enable it only during debugging sessions or in a staging environment.

5. Analyzing Slow Queries

Query Type	Description	Performance Indicators	Tools
Elasticsearch	Slow query detection via `slowlog`	Query execution time exceeding threshold (e.g., 500ms)	`elasticsearch.yml` configuration, `_cat/nodes`
Kibana	Slow query visualization in `Discover` and `Dev Tools`	High latency in query response, long-running aggregations	Kibana UI, `Timelion`, `Canvas`
Logstash	Slow pipeline processing in `logstash.yml`	Backpressure, delayed logs, high queue times	`logstash --config.test_and_exit`, `_node/stats/pipelines`

Example: Configuring Slow Query Logs in Elasticsearch

index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms

index.search.slowlog.threshold.query.logstash.warn: 10s
index.search.slowlog.threshold.query.logstash.info: 5s
index.search.slowlog.threshold.query.logstash.debug: 2s
index.search.slowlog.threshold.query.logstash.trace: 500ms

Example: Identifying Slow Queries in Kibana

{ 
  "query": {
    "bool": {
      "must": [
        { "match": { "status": "error" } },
        { "range": { "timestamp": { "gte": "now-1h" } } }
      ]
    }
  },
  "aggs": {
    "avg_response_time": { "avg": { "field": "response_time" } }
  },
  "size": 0
}

Note: Slow query logs can be configured at the index or cluster level. Use index.search.slowlog.threshold.query.warn to set thresholds for warning-level logging.

Warning: Enabling extensive slow query logging can impact performance and increase disk usage. Monitor logs regularly to avoid log spam.

6. Debugging Shard Allocation

Property	Description	Values
shard allocation	Controls how shards are distributed across nodes	`true` / `false`
allocation.enable	Enables or disables shard allocation	`all`, `primaries`, `new_primaries`, `none`
cluster.routing.allocation.enable	Global setting to enable/disable allocation	`all`, `primaries`, `new_primaries`, `none`

Example: Check shard allocation status

GET /_cluster/allocation/explain

Example: Disable shard allocation

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "none"
  }
}

Example: Enable shard allocation

PUT /_cluster/settings
{
  "transient": {
    "cluster.routing.allocation.enable": "all"
  }
}

Note: Use _cluster/allocation/explain to understand why shards are not being allocated.

Warning: Disabling shard allocation can lead to data unavailability.

7. Resolving Unassigned Shards

Property	Description	Value
unassigned_shards	Number of shards that are not assigned to any node	`0`
shard_status	Status of shards in the cluster	`UNASSIGNED`

Example: Check Unassigned Shards

GET /_cat/shards?v&h=index,shard,prirep,state,unassigned.reason

Example: List Unassigned Shards

GET /_cluster/allocation/explain

Note: Unassigned shards can occur due to node failures, disk space issues, or configuration problems.

Warning: If shards remain unassigned for an extended period, data may become inaccessible.

Reason	Description	Solution
`INDEX_CREATED`	Index was created but shards not allocated yet	Wait for allocation or manually allocate shards
`CLUSTER_RECOVERED`	Cluster recovered from a shutdown or restart	Verify cluster health and wait for recovery
`NODE_LEFT`	Node left the cluster unexpectedly	Reallocate shards to available nodes
`REINITIALIZING`	Shards are being reinitialized after a node restart	Allow time for reinitialization or force allocation

8. Fixing Circuit Breakers

Property	Description	Default Value	Notes
circuit_breaker.enabled	Enables or disables circuit breaker functionality	`true`	Set to `false` to disable circuit breaker checks
circuit_breaker.memory_limit	Maximum memory usage before circuit breaker triggers	`80%`	Can be set as percentage or absolute value (e.g., `2g`)
circuit_breaker.heap_limit	Heap usage threshold for circuit breaker	`95%`	Triggers circuit breaker if heap usage exceeds this limit

Example: Configuring Circuit Breaker in Elasticsearch

elasticsearch:
  circuit_breaker:
    enabled: true
    memory_limit: 80%
    heap_limit: 95%

Note: Circuit breakers prevent out-of-memory errors by monitoring memory usage and stopping operations when thresholds are exceeded.

Warning: Disabling circuit breakers can lead to node crashes or data loss in memory-constrained environments.

Common Fixes for Circuit Breaker Issues

Adjust memory_limit to allow more memory usage
Increase heap size if running on systems with limited RAM
Review and optimize queries that consume large amounts of memory
Use indices.breaker.fielddata.limit to control field data breaker

9. Debugging Mapping Conflicts

Issue	Description	Solution
Mapping Conflict	Occurs when Elasticsearch tries to index a document with a field that has a different data type than the existing mapping.	Use `PUT /{index}/_mapping` to update the mapping or reindex with correct field types.
Dynamic Mapping	When new fields are added to documents, Elasticsearch automatically infers their types. Conflicts arise if the inferred type differs from existing ones.	Disable dynamic mapping or explicitly define field types to prevent automatic type inference.

Example: Handling Mapping Conflict

PUT /my_index/_mapping
{
  "properties": {
    "field_name": {
      "type": "string"
    }
  }
}

Note: When updating mapping, existing documents with conflicting types may not be indexed correctly. Reindexing is often required.

10. Resolving Version Conflicts

Component	Compatible Versions	Notes
Elasticsearch	`8.10.0` - `8.11.0`	Ensure all components are within the same major version range
Logstash	`8.10.0` - `8.11.0`	Must match Elasticsearch version for optimal compatibility
Kibana	`8.10.0` - `8.11.0`	Must be same major version as Elasticsearch and Logstash

Note: Version mismatches can lead to data loss or connectivity issues. Always verify compatibility before upgrading.

Example: Check Component Versions

curl -X GET "localhost:9200/_cluster/health?pretty"

Example: Update Logstash Pipeline

input {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-*"
  }
}

11. Using Hot Threads Analysis

Example: Enable hot threads analysis in Elasticsearch

{
  "cluster": {
    "routing": {
      "allocation": {
        "enable": "true"
      }
    }
  },
  "nodes": {
    "hot_threads": {
      "enabled": "true"
    }
  }
}

Property	Description	Default Value	Notes
nodes.hot_threads.enabled	Enables hot threads analysis	false	Set to true to enable
nodes.hot_threads.interval	How often to check for hot threads	5s	Time duration string (e.g., 10s, 1m)
nodes.hot_threads.max_threads	Maximum number of hot threads to report	3	Integer value
nodes.hot_threads.thread_type	Type of threads to analyze	cpu	Options: cpu, wait, block

Note: Hot threads analysis helps identify performance bottlenecks by monitoring thread usage and identifying threads that consume excessive CPU or are blocked.

Warning: Enabling hot threads analysis can introduce overhead to the cluster, especially in high-load environments.

12. Debugging Ingest Failures

Issue	Description	Solution
Log parsing errors	Errors in parsing logs due to malformed or unexpected log formats.	Check log format configuration in Logstash or Beats. Use `grok` patterns to correctly parse logs.
Indexing failures	Failures in indexing logs into Elasticsearch due to mapping or size issues.	Review index templates and mappings. Ensure field types match log data. Check for oversized documents.
Network connectivity issues	Problems with communication between Beats, Logstash, and Elasticsearch.	Verify network settings, firewall rules, and connectivity. Use `telnet` or `nc` to test port access.
Memory limits exceeded	Logstash or Elasticsearch processes hitting memory limits.	Increase heap size in `jvm.options` or adjust pipeline settings. Monitor memory usage with `top` or `jconsole`.

Example: Logstash Configuration for Debugging

input {
  file {
    path => "/var/log/application.log"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:loglevel} %{GREEDYDATA:message}" }
  }
  date {
    match => [ "timestamp", "ISO8601" ]
  }
}

output {
  elasticsearch {
    hosts => [ "localhost:9200" ]
    index => "app-logs-%{+YYYY.MM.dd}"
  }
  stdout {
    codec => rubydebug
  }
}

Note: Use the stdout output with rubydebug codec to inspect the parsed events before sending them to Elasticsearch.

Warning: Avoid using sincedb_path => "/dev/null" in production environments as it may cause duplicate log entries.