Debugging and Troubleshooting

1. Checking Prometheus Logs

Example: Log levels and configuration

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

# Log level configuration
log:
  level: info
  format: logfmt

# Example of setting log level via CLI
# prometheus --log.level=debug

Log Level	Description	Use Case
debug	Most verbose logging, includes debug messages	Development and detailed troubleshooting
info	Default level, logs informational messages	General monitoring and operational checks
warn	Logs warnings and errors only	Production systems with minimal log overhead
error	Logs only errors	Highly constrained environments or minimal logging

Note: Prometheus logs are written to stderr by default. Use --log.file flag to write logs to a file.

Example: Log file configuration

# Start Prometheus with log file
prometheus --config.file=prometheus.yml \
           --log.file=/var/log/prometheus.log \
           --log.level=debug

Common Log Messages

level=info msg="Loading configuration file" - Configuration loaded successfully
level=warn msg="Scrape failed" - Target scrape issue
level=error msg="Unable to listen on..." - Port binding error
level=debug msg="Querying..." - Debug query execution

2. Using Promtool for Validation

Example: Validate a Prometheus Rule File

promtool check rules /path/to/rules.yml

Command	Description	Flags
promtool check rules	Validates rule files for syntax and semantic errors	`--dry-run` to skip rule evaluation
promtool check metrics	Validates metric names and labels for Prometheus compliance	`--dry-run` to skip metric evaluation
promtool check config	Validates Prometheus configuration files	`--dry-run` to skip config loading

Note: promtool is part of the Prometheus distribution and must be run from the same directory as your config or rules files.

Example: Validate a Prometheus Configuration File

promtool check config prometheus.yml

Example: Validate a Rule File with Custom Rules

groups:
- name: example-group
  rules:
  - alert: HighRequestLatency
    expr: rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m]) > 1
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High request latency"

3. Debugging Scrape Failures

Example: Configuring scrape timeout and interval

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 15s
    scrape_timeout: 10s
    static_configs:
      - targets: ['localhost:9090']

Configuration	Description	Default Value	Notes
scrape_interval	How often to scrape targets	`1m`	Controls how frequently Prometheus scrapes the targets
scrape_timeout	Timeout for scraping a target	`10s`	If exceeded, the scrape is considered failed
scrape_failures	Number of failed scrapes before marking job as down	`10`	Used in conjunction with `scrape_interval`

Note: Use promtool check-config to validate your configuration files.

Warning: Setting scrape_timeout too low can cause false positives in failure detection.

Common Scrape Failure Causes

Network connectivity issues
Incorrect target URLs or ports
Timeouts due to slow response times
Authentication or authorization errors
Invalid metrics format from target

Scrape Failure Troubleshooting Steps

Check Prometheus logs for error messages related to the job
Verify target endpoint is accessible via browser or curl
Ensure scrape_interval and scrape_timeout are set appropriately
Validate that the target endpoint returns valid Prometheus metrics
Use promtool check-metrics to validate metric format

4. Investigating Target Status

Field	Description	Example
job_name	Name of the job that discovered this target	`node_exporter`
instance	Unique identifier for the target (usually IP:port)	`192.168.1.10:9100`
scrape_health	Current health status of scraping (up, down, or error)	`up`
last_scrape	Timestamp of last successful scrape	`2024-05-20T12:30:00Z`
scrape_duration_seconds	Duration in seconds for last scrape	`0.05`

Example: Querying target status via Prometheus API

curl -X GET http://prometheus-server:9090/api/v1/targets

Example: Filtering targets by job name

curl -X GET 'http://prometheus-server:9090/api/v1/targets?job=node_exporter'

Note: The scrape_health field can be up, down, or error. If a target is down, it means Prometheus cannot scrape the endpoint. If it's error, there was an error during scraping.

Common Target Status Issues

Down: Prometheus cannot connect to the target endpoint
Error: Scraping failed due to configuration or timeout issues
Up: Target is reachable and scraping is successful

5. Debugging Relabeling

Option	Description	Notes
action	`replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`	Defines the action to perform on the label
source_labels	List of label names to read from	Used in actions like `replace`, `hashmod`
separator	Separator for combining labels	Defaults to `;`
target_label	Label name to write to	Required for `replace`, `hashmod`
regex	Regular expression to match source labels	Used in `replace`, `keep`, `drop`
replacement	Replacement string for `replace`	Defaults to `$1$2...` if not specified

Example: Relabeling Configuration

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    regex: (?i)true
    action: keep
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    target_label: __metrics_path__
    regex: (.+)
    replacement: ${1}

Note: Use relabel_configs in Prometheus configuration to debug label transformations.

Debugging Steps

Check scrape_configs for relabeling rules
Use promtool check-config to validate configuration
Inspect metrics in Prometheus UI for label values
Use relabel_configs with keep or drop to filter targets
Verify regex patterns match expected label values

Common Relabeling Actions

keep: Keep only metrics matching a regex
drop: Drop metrics matching a regex
replace: Replace label values using regex capture groups
hashmod: Apply hash modulo to labels

6. Using Query Inspector

Example: Inspecting a Query in Prometheus

rate(http_requests_total[5m])

Query Component	Description	Usage
rate()	Calculates the per-second average rate of increase of a counter	`rate(http_requests_total[5m])`
[5m]	Time range for the rate calculation	`rate(http_requests_total[5m])`

Note: The Query Inspector is accessible in the Prometheus web UI under the "Graph" tab.

Steps to Use Query Inspector

Navigate to the Prometheus web UI
Go to the "Graph" tab
Enter your query in the input field
Click "Execute" to run the query
Use the "Query Inspector" button to analyze query details

Query Inspector vs Direct Query

Query Inspector: Provides detailed breakdown of query execution
Direct Query: Returns raw metric values

7. Debugging Missing Metrics

Example: Checking Metric Exposure

curl -s http://localhost:9090/metrics | grep -E &#34;^myapp_&#34;

Step	Action	Description
1	`curl -s http://localhost:9090/metrics`	Fetch all exposed metrics from the Prometheus endpoint
2	`grep -E "^myapp_"`	Filter for metrics related to your application
3	Check output	Verify that expected metrics are present in the response

Note: Ensure the application exposing metrics is running and configured correctly.

Warning: If no metrics are returned, check the application's metric endpoint and Prometheus configuration.

Common Causes of Missing Metrics

Application not exposing metrics at the configured endpoint
Prometheus server misconfigured to scrape from wrong address/port
Metrics not properly registered or exported in the application code
Network issues preventing access to the metrics endpoint

8. Investigating High Memory Usage

Issue	Description	Solution
High memory usage in Prometheus server	Memory consumption increases over time, leading to out-of-memory errors.	Monitor memory metrics using `go_memstats_alloc_bytes` and `process_resident_memory_bytes`.
Memory spikes during queries	Queries cause temporary memory spikes due to large result sets or complex operations.	Use `query_duration_seconds` to identify slow queries and optimize them with `max_concurrent_queries`.

Example: Memory Metrics Query

go_memstats_alloc_bytes
process_resident_memory_bytes

Example: Query Duration Analysis

rate(prometheus_engine_query_duration_seconds_sum[5m]) / rate(prometheus_engine_query_duration_seconds_count[5m])

Note: Memory usage can be influenced by the number of time series and the retention policy. Adjust --storage.tsdb.retention.time to manage memory usage effectively.

Warning: High memory usage can lead to Prometheus server crashes. Ensure sufficient system resources and consider using --storage.tsdb.wal-compression to reduce memory overhead.

Configuration Option	Description	Default Value	Impact
--storage.tsdb.wal-compression	Enables WAL compression to reduce memory usage.	false	Reduces memory consumption during query processing.
--query.max-concurrent	Limits concurrent queries to prevent memory exhaustion.	10	Prevents high memory usage from concurrent query execution.

9. Debugging Alert Rules

Property	Description	Example
alertname	Name of the alert rule	`InstanceDown`
job	Job name from the scrape target	`node_exporter`
instance	Instance identifier	`node01:9100`
severity	Severity level of the alert	`critical`

Example: Alert Rule Definition

groups:
- name: node
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: "Instance {{ $labels.instance }} down"

Example: Querying Alert State

curl -X GET http://localhost:9090/api/v1/alerts

Note: Use the Prometheus web UI or API to inspect active alerts and their states.

Workflow: Debugging Alert Rule Issues

Check if the alert rule expression evaluates to true for the intended conditions
Verify that the rule is correctly defined in the configuration file
Ensure the Prometheus server has loaded the updated configuration
Use the /api/v1/alerts endpoint to inspect alert states
Review logs for any errors related to rule evaluation or scraping

10. Using TSDB Status

Example: Querying TSDB Status

curl -X GET http://localhost:9090/api/v1/status/tsdb

Field	Description	Values
build	Build information	`{version, revision, branch, buildUser, buildDate}`
config	Configuration details	`{...` (see Prometheus config docs)
flags	Command-line flags	`{...` (see Prometheus flags)
runtimeInfo	Runtime information	`{startTime, goroutineCount}`
tsdb	TSDB-specific status	`{...}` (see below)

Note: The TSDB status endpoint provides low-level details about the storage engine's internal state, useful for advanced debugging and performance tuning.

TSDB Subfield	Description	Values
head	Current head block info	`{baseTime, size, minTime, maxTime}`
blocks	List of block metadata	`[{minTime, maxTime, size, numSamples}]`
compaction	Compaction status	`{running, failed, lastDuration}`
wal	Write-ahead log status	`{size, lastSegment, firstSegment}`

11. Checking Recording Rule Evaluation

Field	Description	Example
rule_name	Name of the recording rule	`node_cpu_seconds_total:rate1m`
evaluation_time	Time taken to evaluate the rule	`12ms`
last_evaluation	Timestamp of last evaluation	`2024-05-15T10:30:00Z`
state	Current state of the rule (active, inactive)	`active`

Example: Querying Recording Rule Metrics

prometheus_rule_evaluations_total{rule_name="node_cpu_seconds_total:rate1m"}

Example: Checking Rule Evaluation Duration

prometheus_rule_evaluation_duration_seconds{rule_name="node_cpu_seconds_total:rate1m"}

Note: Use prometheus_rule_evaluations_total to check how many times a rule has been evaluated and prometheus_rule_evaluation_duration_seconds to monitor evaluation performance.

12. Debugging Federation Issues

Issue	Description	Solution
Federated metrics not appearing	Metrics from federated targets are missing in the federation endpoint.	Verify that the target Prometheus instance is configured with `enable_federation: true` and that the federation URL matches the configured endpoint.
Incorrect metric names or labels	Federated metrics have unexpected names or labels due to label relabeling.	Check the `relabel_configs` in the federation configuration to ensure labels are correctly handled.
Timeouts during federation	Metrics are not fetched due to timeout errors from the federated instance.	Adjust `scrape_timeout` and `scrape_interval` in the federation configuration to allow more time for scraping.
Authentication failures	Access to federated Prometheus instances is denied due to missing or incorrect credentials.	Ensure that the `basic_auth` or `authorization` headers are correctly configured in the federation job.

Example: Federation Configuration

scrape_configs:
  - job_name: "federate"
    scrape_interval: 15s
    honor_labels: true
    metrics_path: /federate
    params:
      match[]:
        - "{job='prometheus'}"
        - "{__name__=~'job:.*'}"
    static_configs:
      - targets:
          - "prometheus-remote:9090"
    basic_auth:
      username: "federate_user"
      password: "secure_password"

Note: The honor_labels setting ensures that labels from the federated instance are preserved, which is critical for accurate metric aggregation.

Warning: Federation can lead to high cardinality if not carefully configured. Avoid matching all metrics without filtering.