Troubleshooting Common Issues

1. Resolving Out of Memory Errors

Issue	Description	Solution
High memory usage	Excessive memory consumption due to large time series or high cardinality metrics.	Use `scrape_interval` and `scrape_timeout` to reduce load. Implement `metric_relabel_configs` to drop unnecessary metrics.
Memory spikes during scraping	Temporary memory increase when scraping large numbers of targets.	Adjust `scrape_timeout` to prevent long waits. Use `concurrent_scrape_limit` to limit concurrent scrapes.
Memory leaks in remote storage	Memory consumption increases over time when using remote write.	Monitor `prometheus_remote_storage_queue_length` and `prometheus_remote_storage_samples_in_total`. Adjust `remote_write` configuration to reduce queue size.

Example: Adjusting scrape settings to reduce memory usage

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    static_configs:
      - targets: [localhost:9090]

Example: Using metric relabeling to drop high-cardinality metrics

scrape_configs:
  - job_name: "app"
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "app_.*"
        action: drop

Note: High cardinality metrics can cause memory issues. Use prometheus_tsdb_wal_truncations_total to monitor WAL truncation and prometheus_tsdb_head_chunks to check head chunks.

Warning: Reducing scrape_interval too aggressively can cause data loss or staleness in metrics.

2. Fixing Failed Scrapes

Issue	Description	Solution
HTTP 404 Not Found	Target endpoint returns 404 status code.	Verify the scrape target URL and ensure the endpoint exists.
HTTP 401 Unauthorized	Authentication required but not provided.	Configure basic auth or bearer token in Prometheus config.
Timeout Error	Target does not respond within scrape timeout.	Increase `scrape_timeout` or optimize target performance.
Connection Refused	Target server is unreachable or not running.	Check network connectivity and service status on target host.
Invalid Metrics Format	Metrics returned do not conform to Prometheus format.	Validate metrics output using `promtool check metrics`.

Example: Configuring Basic Auth for Scraping

scrape_configs:
  - job_name: 'myapp'
    metrics_path: /metrics
    scrape_interval: 15s
    basic_auth:
      username: "prometheus"
      password: "secret"

Note: Ensure that the password is properly escaped or stored securely (e.g., using secrets management).

Warning: Using plaintext passwords in configuration files is insecure. Prefer using file_sd_configs or external secret stores.

3. Debugging Missing Metrics

Example: Checking Metric Exposure

curl -s http://localhost:9090/metrics | grep -E &#34;^my_metric_name&#34;

Step	Action	Description
1	Check Prometheus target status	Navigate to `/targets` in the Prometheus UI to verify that your service is scraping.
2	Verify metric endpoint	Ensure the application exposes a `/metrics` endpoint and it's accessible from Prometheus.
3	Validate metric format	Metrics must follow the Prometheus exposition format with proper labels and types.
4	Check for errors in logs	Review Prometheus and application logs for scraping or parsing errors.

Note: Use promtool check metrics to validate metric format locally.

Common Causes of Missing Metrics

Incorrect endpoint: Service not exposing /metrics
Network issues: Prometheus cannot reach the target
Label mismatch: Metrics not labeled consistently across instances
Scrape timeout: Target takes longer than scrape interval to respond
Invalid metric name: Contains unsupported characters or exceeds length limits

4. Resolving High Cardinality

Problem	Solution	Notes
High metric cardinality	Use label relabeling to reduce unique label combinations	High cardinality can cause performance issues and increased memory usage
Unbounded labels	Implement label value limits or regex filtering	Example: `relabel_configs:` with `source_labels` and `regex`
Dynamic metric names	Use `metric_relabel_configs` to normalize metric names	Prevent creation of new metrics with unique names

Example: Relabeling to Reduce Cardinality

relabel_configs:
  - source_labels: [__name__]
    regex: (http_requests_total|http_response_time)
    target_label: job
  - source_labels: [instance]
    regex: (.+)
    target_label: instance_short
    replacement: ${1}

Note: Label relabeling is a powerful feature to reduce high cardinality without changing the actual metrics collected.

5. Fixing Slow Queries

Problem	Symptoms	Solution
High query latency	Queries taking more than 5 seconds to complete	Use `rate()` with appropriate window size and avoid `max_over_time()` or `min_over_time()` on large datasets
Memory usage spikes	High memory consumption during query execution	Use `max_concurrent_queries` to limit concurrent queries and set `query_timeout` to prevent long-running queries
Querying large time ranges	Slow response when querying large historical data	Break down queries into smaller time windows or use `range_vector_selector` with optimized step sizes

Example: Optimizing a Slow Query


rate(http_requests_total[5m])

Note: Use rate() instead of increase() to avoid overcounting in high-cardinality scenarios.

Warning: Avoid using max_over_time() on large datasets as it can cause memory exhaustion.

6. Debugging WAL Corruption

WAL (Write-Ahead Log) corruption in Prometheus can lead to data loss or inconsistent metrics. Debugging WAL corruption involves identifying the root cause and taking corrective actions.

Step	Action	Description
1	Check logs for WAL errors	Look for error messages related to WAL in Prometheus logs. Common messages include `WAL corruption detected` or `corrupted block`.
2	Verify disk health	Corruption can stem from hardware issues. Use tools like `smartctl` to check disk health and ensure no I/O errors.
3	Check for power outages or system crashes	Ungraceful shutdowns can cause WAL to become corrupted. Review system logs for unexpected shutdowns or power loss.
4	Validate WAL files manually	Use `promtool` to inspect WAL files for corruption. Run: `promtool check-wal /path/to/wal/directory`.
5	Restart Prometheus with recovery mode	Start Prometheus in recovery mode to attempt automatic repair of corrupted blocks. Use the `--storage.tsdb.wal-replay-skip` flag to skip problematic blocks.

Example: Using promtool to check WAL

promtool check-wal /prometheus/data/wal

Note: If WAL corruption is severe and recovery fails, you may need to restore from backups or re-ingest data.

Warning: Never attempt to manually edit or delete WAL files without a full backup. This can lead to further data loss.

7. Resolving Disk Space Issues

Issue	Description	Solution
High disk usage by Prometheus	Prometheus stores time-series data on disk, which can grow rapidly if not managed properly.	Use `retention.time` configuration to limit data storage duration. Example: `storage.tsdb.retention.time = 30d`
Disk space exhaustion	When Prometheus runs out of disk space, it stops functioning and may cause data loss.	Monitor disk usage with tools like `df -h`. Set up alerts for low disk space using Prometheus itself or external monitoring tools.
TSDB storage growth	The Time Series Database (TSDB) can grow without limits if retention policies are not configured.	Configure `storage.tsdb.max-block-duration` and `storage.tsdb.min-block-duration` to control block size and retention.

Example: Configuring Retention Policy

storage:
  tsdb:
    retention.time: 30d
    max-block-duration: 2h
    min-block-duration: 1h

Note: Adjust retention time based on your data needs and available storage capacity.

Warning: Setting retention too low may cause loss of historical data needed for analysis.

8. Fixing Rule Evaluation Failures

Example: Alert Rule with Syntax Error

# Incorrect rule with syntax error
- alert: HighMemoryUsage
  expr: memory_usage > 80
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "High memory usage detected"
    description: "Memory usage is above 80% for more than 5 minutes"

Field	Description	Notes
alert	Name of the alert rule	Must be unique within the group
expr	PromQL expression to evaluate	Must return a vector or scalar
for	Duration to wait before firing	Optional, defaults to 0s
labels	Additional labels to attach to alerts	Used for grouping and routing
annotations	Additional metadata for alerts	Used for descriptions and summaries

Note: Ensure all PromQL expressions are valid and return the expected data types. Use promtool check rules to validate rule files before applying them.

Example: Valid Alert Rule

# Corrected rule with proper syntax
- alert: HighMemoryUsage
  expr: instance_memory_usage_bytes / instance_memory_total_bytes * 100 > 80
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "High memory usage detected"
    description: "Memory usage is above 80% for more than 5 minutes"

Common Causes of Rule Evaluation Failures

Invalid PromQL Expression: Syntax errors or unsupported functions
Missing Metrics: Querying metrics that don't exist or are not scraped
Time Range Issues: Using time ranges that don't align with data availability
Label Mismatch: Inconsistent label values across time series
Rule File Format: YAML syntax errors or malformed rule definitions

Steps to Debug Rule Evaluation Failures

Validate rule file syntax using promtool check rules
Check Prometheus logs for error messages related to rule evaluation
Test PromQL expressions in the Prometheus web UI or console
Verify that all metrics referenced in rules exist and are being scraped
Ensure labels used in rules match those present on the time series

9. Debugging Remote Write Failures

Example: Configuring remote write with debug logging

remote_write:
  - url: http://prometheus-remote-write:9090/api/v1/write
    queue_config:
      capacity: 1000
      max_shards: 10
    send_exemplars: true
    headers:
      Authorization: "Bearer your-token-here"

Field	Description	Notes
url	Remote write endpoint URL	Mandatory field for remote write
queue_config	Configuration for the queue of samples to send	Controls buffering and concurrency
capacity	Maximum number of samples in the queue	Defaults to 1000
max_shards	Maximum number of concurrent shards	Defaults to 10
send_exemplars	Whether to send exemplars	Enabled by default in newer versions
headers	HTTP headers to send with requests	Useful for authentication

Steps to Debug Remote Write Failures

Check Prometheus logs for error messages related to remote write
Verify the remote endpoint is reachable and responding correctly
Ensure the URL in remote_write configuration is correct
Review queue_config settings for capacity and shard limits
Check if the remote endpoint supports the required format (e.g., protobuf)

Common Remote Write Issues

Connection timeout: Increase queue_config capacity or reduce max_shards
Authentication failure: Check headers and token validity
Rate limiting: Reduce the rate of samples sent or increase queue size
Unsupported format: Ensure remote endpoint supports Prometheus's protobuf format

Note: Remote write failures can also stem from misconfigured remote endpoints or network issues between Prometheus and the remote system.

10. Resolving Target Discovery Issues

Issue	Possible Cause	Solution
Targets not appearing in UI	Target not registered with service discovery or scrape config misconfigured	Check Prometheus config for correct `scrape_configs` and ensure targets are reachable via the configured endpoints
Target state is "down"	Scrape timeout, incorrect endpoint, or target unreachable	Verify target endpoint is accessible and Prometheus can scrape it within the configured timeout. Use `curl` to test connectivity.
Discovery fails due to DNS resolution	Incorrect or missing DNS entries for target hosts	Ensure DNS resolution works for target hostnames. Use `dig` or `nslookup` to verify DNS entries.
Targets not discovered in Kubernetes	Missing or incorrect service discovery configuration for Kubernetes	Verify Prometheus is configured with `kubernetes_sd_configs` and has proper RBAC permissions to access Kubernetes API.

Example: Kubernetes SD Config

scrape_configs:
- job_name: "kubernetes-pods"
  kubernetes_sd_configs:
  - role: pod
    api_server: https://kubernetes.default.svc
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true

Note: Ensure that the kubernetes_sd_configs block is correctly configured with appropriate permissions and that the target pods have the correct annotations for scraping.

11. Fixing Alert Routing Problems

Problem	Symptoms	Solution
Misconfigured receiver	Alerts not reaching the intended notification channel (e.g., Slack, email)	Verify receiver configuration in `prometheus.yml` or `alertmanager.yml`. Ensure the receiver name matches in both files.
Incorrect route matching	Alerts are not being routed to the correct receiver due to label mismatches	Check `route.match` and `route.match_re` in the Alertmanager config. Use regex patterns for flexible matching.
Missing or invalid labels	Alerts are not being grouped or routed properly due to missing labels	Add required labels to alerts (e.g., `severity`, `team`) using `labels` in rules or Prometheus relabeling.
Duplicate route configuration	Alerts may be routed to multiple receivers or duplicated	Ensure each alert matches only one route by using `continue: false` in parent routes to prevent fall-through.

Example: Alertmanager Route Configuration

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: team-slack
  routes:
  - match:
      severity: critical
    receiver: critical-slack
    continue: true
  - match:
      severity: warning
    receiver: team-slack

receivers:
- name: critical-slack
  slack_configs:
  - api_url: &#34;https://hooks.slack.com/services/...&#34;
    channel: &#34;#critical-alerts&#34;

- name: team-slack
  slack_configs:
  - api_url: &#34;https://hooks.slack.com/services/...&#34;
    channel: &#34;#team-alerts&#34;

Note: Use continue: true to allow alerts to match additional routes, and false to stop matching after the first route.

Warning: Misconfigured routes can cause alerts to be dropped or duplicated. Always test alert routing in a non-production environment.

12. Debugging TLS Connection Issues

Issue	Description	Solution
Certificate verification fails	Server certificate is not trusted due to missing CA or incorrect hostname	Ensure the Prometheus server has access to the correct CA certificate and that the target endpoint's hostname matches the certificate
Insecure skip verify enabled	Using `insecure_skip_verify: true` can lead to man-in-the-middle attacks	Use proper certificate validation instead of skipping verification
Outdated TLS version	Server or client uses an outdated TLS version (e.g., TLS 1.0)	Update to TLS 1.2 or higher for secure connections

Example: TLS Configuration in Prometheus

scrape_configs:
  - job_name: "secure-target"
    metrics_path: /metrics
    scheme: https
    tls_config:
      ca_file: /etc/ssl/certs/ca.crt
      cert_file: /etc/ssl/certs/client.crt
      key_file: /etc/ssl/private/client.key
      server_name: target.example.com

Note: When debugging TLS issues, use tools like openssl s_client or curl --verbose to test the connection outside of Prometheus.

Warning: Avoid using insecure_skip_verify: true in production environments as it disables certificate validation.