Implementing Cardinality Management

1. Monitoring Series Count

Property	Description	Default Value	Notes
prometheus_tsdb_head_series	Current number of active series in the head chunk	N/A	Exposes the total number of time series in Prometheus' head chunk, which is a key indicator for cardinality
prometheus_tsdb_head_chunks	Current number of chunks in the head chunk	N/A	Helps understand how many chunks are currently stored in memory
prometheus_tsdb_wal_truncations_total	Total number of WAL truncations	N/A	Indicates how often the Write-Ahead Log (WAL) has been truncated, which can affect series count

Example: Querying Series Count

prometheus_tsdb_head_series

Note: The prometheus_tsdb_head_series metric is essential for monitoring the cardinality of your Prometheus instance. High series counts can lead to performance degradation and increased memory usage.

2. Identifying High-Cardinality Labels

Label Name	Description	Cardinality Risk	Example
user_id	Unique identifier for each user	High	`user_id="123456789"`
session_id	Unique session identifier	High	`session_id="abc123def456"`
request_id	Unique request identifier	High	`request_id="req-001"`
ip_address	IP address of the client	Medium	`ip_address="192.168.1.1"`
timestamp	Time of the event	High	`timestamp="1640995200"`

Example: Monitoring label cardinality in Prometheus

prometheus:
  scrape_configs:
    - job_name: "app"
      static_configs:
        - targets: [ "localhost:8080" ]
      metric_relabel_configs:
        - source_labels: [ "__name__" ]
          regex: ".*_count"
          target_label: "metric_type"
          replacement: "counter"

Note: High cardinality labels can lead to increased memory usage and slower query performance in Prometheus.

Best Practices for Label Cardinality

Avoid using labels with unique values like IDs or timestamps
Use enums or predefined values for labels where possible
Monitor label cardinality using prometheus_tsdb_head_series
Implement label relabeling to reduce unnecessary labels

3. Using Relabeling to Drop Labels

Prometheus provides the relabel_config mechanism to modify or drop labels during metric ingestion. This is particularly useful for managing label cardinality by removing or renaming labels that contribute to high cardinality.

Field	Description	Example
action	The relabeling action to perform. Valid values are `replace`, `keep`, `drop`, `hashmod`, and `labelmap`.	`drop`
source_labels	List of label names to use as input for the relabeling action.	`[__name__, instance]`
target_label	The label name to modify or create.	`job`
regex	Regular expression to match the source labels.	`.*`
replacement	Replacement string for `replace` actions.	`$1`

Example: Dropping Unwanted Labels

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ['localhost:9100']
    relabel_configs:
      - source_labels: [__meta_consul_service_id]
        target_label: service_id
        action: replace
      - source_labels: [__meta_consul_service_id]
        target_label: service_id
        action: drop

Note: The drop action removes labels entirely, which can significantly reduce label cardinality in metrics.

Warning: Be cautious when dropping labels, as it may impact the ability to filter or aggregate metrics effectively.

4. Implementing Label Limits

Example: Configuring label limits in Prometheus

global:
  evaluation_interval: 1m
  scrape_interval: 15s

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: [ "localhost:9090" ]
    metric_relabel_configs:
      - source_labels: [ __name__ ]
        regex: ".*"
        target_label: "__name__"
        action: "replace"

    # Label limit configuration
    metric_relabel_configs:
      - source_labels: [ __name__ ]
        regex: ".*"
        target_label: "__name__"
        action: "replace"
        replacement: "${1}"

    # Example of limiting label cardinality by dropping labels
    metric_relabel_configs:
      - source_labels: [ __name__ ]
        regex: "http_requests_total"
        target_label: "method"
        action: "keep"
      - source_labels: [ __name__ ]
        regex: "http_requests_total"
        target_label: "status"
        action: "keep"
      - source_labels: [ __name__ ]
        regex: "http_requests_total"
        target_label: "endpoint"
        action: "drop"

Configuration Option	Description	Default Value	Notes
max_label_name_length	Maximum allowed length for label names	`128`	Prevents excessively long label names that could cause performance issues
max_label_value_length	Maximum allowed length for label values	`1024`	Helps prevent memory bloat from very long label values
max_metric_length	Maximum allowed length for metric names	`1024`	Prevents overly long metric names that may impact storage or queries
max_samples_per_query	Maximum number of samples a single query can return	`1000000`	Prevents queries from consuming too many resources

Note: These limits are enforced at scrape time and can be adjusted via the Prometheus configuration file.

Warning: Setting these limits too low can cause issues with metric ingestion or query execution. Ensure that your metrics and labels remain within the defined constraints.

5. Configuring Sample Limits

Option	Values	Notes
limit_samples_per_query	`1000000`	Maximum number of samples a single query can return
limit_samples_per_metric	`100000`	Maximum number of samples per metric in a single query
limit_metric_name_length	`1024`	Maximum length of metric names

Example: Configuring sample limits in Prometheus configuration

global:
  evaluation_interval: 1m
  scrape_interval: 15s

rule_files:
  - alerts.yml

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

limit_config:
  max_samples_per_query: 1000000
  max_samples_per_metric: 100000
  max_metric_name_length: 1024

6. Using Metric Relabeling

Prometheus metric relabeling allows you to modify metric labels before they are ingested or stored. This is useful for controlling label cardinality, renaming labels, or filtering out unwanted metrics.

Configuration	Description	Example
action	The relabeling action to perform. Valid actions are `replace`, `keep`, `drop`, `hashmod`, and `labelmap`.	`action: replace`
source_labels	List of label names to use as input for the relabeling action.	`source_labels: [__name__, instance]`
target_label	The label name to modify or create.	`target_label: job`
regex	Regular expression to match the source labels.	`regex: (.)(\..)`
replacement	Replacement string for `replace` actions. Supports regex capture groups.	`replacement: $1`

Example: Rename a Label

relabel_configs:
  - source_labels: [job]
    target_label: application
    action: replace

Example: Drop Metrics Based on Label Value

relabel_configs:
  - source_labels: [__name__]
    regex: 'go_.*'
    action: drop

Note: Relabeling rules are processed in order, and each rule can affect subsequent rules. Use keep or drop actions carefully to avoid unintended metric filtering.

Warning: Improperly configured relabeling can lead to high label cardinality or loss of important metrics. Always test configurations in a non-production environment.

7. Dropping Unwanted Metrics

When metrics are no longer needed or are causing cardinality issues, they can be dropped using various mechanisms in Prometheus.

Metric	Drop Method	Description
metric_name	`drop_metric`	Remove metric from scrape targets or use relabeling rules to drop it

Example: Using Relabeling to Drop Metrics

scrape_configs:
  - job_name: 'myapp'
    relabel_configs:
      - source_labels: [__name__]
        regex: 'old_metric_.*'
        action: drop

Note: Relabeling rules are processed before metrics are stored, so dropping happens at scrape time.

8. Implementing Target Relabeling

Target relabeling in Prometheus allows you to modify the labels of scraped metrics before they are stored. This is useful for normalizing labels, filtering unwanted metrics, or enriching data.

Field	Description	Example
action	The relabeling action to perform. Valid actions are `replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`.	`replace`
source_labels	List of label names to use as input for the relabeling action.	`[__address__]`
separator	Separator to use when joining multiple source labels.	`;`
target_label	Label name to write the result to. Required for `replace`, `hashmod`, and `labelmap`.	`instance`
regex	Regular expression to match the input labels. Used in `replace`, `keep`, and `drop`.	`(.):(.)`
replacement	Replacement string for `replace` actions. Supports regex capture groups.	`$1:$2`

Example: Replace Address Label

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ['localhost:9100']
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: (.*):(.*) 
        replacement: $1:$2

Example: Drop Metrics Based on Label

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ['localhost:9100']
    relabel_configs:
      - source_labels: [__name__]
        regex: "node_.*"
        action: drop

9. Monitoring Cardinality Growth

Metric	Description	Usage
prometheus_tsdb_head_series	Number of series in the head block	Monitor for sudden increases indicating high cardinality
prometheus_tsdb_head_chunks	Number of chunks in the head block	High chunk count may indicate high cardinality or frequent updates
prometheus_tsdb_wal_truncations	Number of WAL truncations	Indicates how often the WAL is truncated, can be related to high cardinality

Example: Alerting on High Cardinality

groups:
- name: cardinality-alerts
  rules:
  - alert: HighCardinalitySeries
    expr: prometheus_tsdb_head_series > 100000
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High cardinality detected"
      description: "The number of series in the head block has exceeded 100,000."

Note: Regular monitoring of these metrics helps in identifying and mitigating potential cardinality issues before they impact system performance.

10. Using TSDB Status Page

Example: Accessing TSDB Status Page

curl http://localhost:9090/status/tsdb

Field	Description	Notes
head	Information about the head chunk	Includes size and memory usage
series	Number of series in the database	Used to monitor cardinality growth
chunks	Total number of chunks stored	Indicates storage efficiency
memory	Memory usage in bytes	Helps identify resource consumption

Note: The TSDB status page provides real-time metrics about the internal state of the Prometheus storage engine.

11. Optimizing Label Design

Label Name	Description	Best Practice	Notes
instance	Identifies the target instance	Include for multi-instance setups	Default label added by Prometheus
job	Identifies the job that scraped the target	Always include for clarity	Default label added by Prometheus
status_code	HTTP status code of the response	Use for HTTP metrics	Can lead to high cardinality if not controlled
method	HTTP method used (GET, POST, etc.)	Use for HTTP metrics	May increase label cardinality
endpoint	HTTP endpoint being accessed	Use for detailed endpoint metrics	High cardinality risk if not limited

Example: Label Design for HTTP Metrics

http_requests_total
  {method="GET", endpoint="/api/users", status_code="200"}

Note: Avoid using labels with high cardinality values like user IDs or session tokens.

Warning: High label cardinality can cause performance issues and increased memory usage in Prometheus.

Cardinality Risk	Impact	Recommendation
`user_id`	Very high	Avoid using as a label
`session_token`	Very high	Avoid using as a label
`endpoint`	High	Use with caution; consider grouping or regex matching
`status_code`	Medium	Use with filtering or aggregation

12. Avoiding Cardinality Explosions

Example: Label values with high cardinality

prometheus:
  scrape_configs:
    - job_name: "app"
      static_configs:
        - targets: ["localhost:8080"]
      metric_relabel_configs:
        - source_labels: [instance]
          target_label: instance
          regex: "(.*)"
          replacement: "${1}"

Technique	Description	Impact
Label cardinality limits	Set limits on the number of unique label values per metric	Prevents explosion of time series
Metric relabeling	Filter or drop high-cardinality labels before ingestion	Reduces storage and query overhead
Sample rate limiting	Reduce the frequency of metric collection for high-cardinality series	Decreases data volume and resource usage

Note: High cardinality labels can lead to performance issues and increased storage costs in Prometheus.

Warning: Dropping labels too aggressively can cause data loss or incorrect aggregations.

Best Practices

Monitor label cardinality regularly using prometheus_tsdb_wal_truncations_total
Use metric_relabel_configs to filter or drop labels with high cardinality
Implement rate limiting for metrics that are expected to have many unique values