Working with Relabeling

1. Understanding Relabel Actions

Action	Description	Usage	Notes
replace	Replace the label value with a regex capture group	`action: replace`	Useful for transforming label values based on patterns
keep	Keep only metrics that match the label value	`action: keep`	Filters out metrics that do not match the regex
drop	Drop metrics that match the label value	`action: drop`	Filters out metrics that match the regex
hashmod	Hash the label value and take modulo to assign new label value	`action: hashmod`	Useful for load balancing or sharding
labelmap	Map labels to new names using regex	`action: labelmap`	Applies regex to all labels and renames them
labeldrop	Drop labels matching a regex pattern	`action: labeldrop`	Removes labels that match the regex pattern
labelkeep	Keep only labels matching a regex pattern	`action: labelkeep`	Retains only labels that match the regex pattern

Example: Replace action usage

relabel_configs:
  - source_labels: [__name__]
    regex: '(.+)_total'
    target_label: metric_name
    replacement: '${1}'
    action: replace

Example: Keep action usage

relabel_configs:
  - source_labels: [job]
    regex: 'production.*'
    action: keep

Example: Drop action usage

relabel_configs:
  - source_labels: [job]
    regex: 'staging.*'
    action: drop

2. Using source_labels

Property	Description	Example
source_labels	List of label names to extract values from. Values are concatenated with the separator.	`source_labels: ["__address__", "__meta_kubernetes_pod_name"]`
separator	Separator to use when concatenating values from `source_labels`.	`separator: ":"`
target_label	Label name to write the concatenated value to.	`target_label: "instance"`
regex	Regular expression to match and capture groups from the concatenated value.	`regex: "(.+):(.+)"`
replacement	Replacement string for matched regex groups. Use `$1`, `$2` etc.	`replacement: "${1}-${2}"`

Example: Relabeling with source_labels

relabel_configs:
  - source_labels: [__address__, __meta_kubernetes_pod_name]
    separator: :
    target_label: instance
    regex: (.+):(.+)
    replacement: ${1}-${2}

Note: The source_labels field is required for relabeling rules that extract values from existing labels.

3. Configuring target_label

Property	Description	Default	Notes
target_label	Label name to which the value of `replacement` is assigned.	None	Required field when using relabeling rules.

Example: Basic target_label configuration

relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    regex: true
    target_label: scrape
    replacement: "true"

Note: The target_label must be a valid Prometheus label name. It can contain alphanumeric characters, underscores, and colons.

4. Setting Regex Patterns

Pattern Type	Description	Example	Notes
source_labels	Labels to match against	`__name__`	Used to define the input labels for regex matching
regex	Regular expression to match against source labels	`(.*)`	Matches any string, capturing all characters in group 1
target_label	Label to write the result to	`job`	If not specified, the result is written to the metric name
replacement	String to replace matched groups with	`$1`	Uses captured groups from the regex pattern

Example: Relabeling with Regex

relabel_configs:
  - source_labels: [__name__]
    regex: (.*)
    target_label: job
    replacement: $1

Note: The regex pattern must be a valid Go-style regular expression. Capture groups are referenced using $1, $2, etc.

5. Using Replacement Strings

Replacement strings in Prometheus relabeling allow you to dynamically construct new label values using regex capture groups. They are specified in the replacement field of a relabel configuration.

Field	Description	Example
replacement	Template string for constructing new label values. Supports `$1`, `$2`, etc., for regex capture groups.	`server-$1`

Example: Using Capture Groups in Replacement

relabel_configs:
  - source_labels: [__address__]
    regex: '([^:]+):.*'
    target_label: instance
    replacement: 'server-$1'

Note: The $1, $2, etc., refer to capture groups defined in the regex field. If a capture group is not matched, it resolves to an empty string.

6. Implementing Keep Action

Property	Value	Description
action	`keep`	Keeps metrics that match the specified regex pattern
source_labels	`__name__`	Label(s) to match against the regex pattern
regex	`^metrics_.*`	Regular expression to match metric names

Example: Keep action configuration

relabel_configs:
  - source_labels: [__name__]
    regex: ^metrics_.*
    action: keep

7. Implementing Drop Action

Property	Description	Example
action	Specifies the relabeling action to perform. For dropping metrics, use `drop`.	`action: drop`
source_labels	List of label names to use as input for the relabeling operation.	`source_labels: [job]`
regex	Regular expression to match the source label values. If a match is found, the metric is dropped.	`regex: (.job.api.*)`
replacement	Value to replace the matched label with (optional). Not used in drop actions.	`replacement: $1`

Example: Drop Metrics Based on Label Value

relabel_configs:
  - source_labels: [job]
    regex: (.*job.*api.*)
    action: drop

Note: The drop action removes metrics entirely if the regex matches. It is useful for filtering out unwanted metrics from scraping.

8. Using labelmap Action

Field	Description	Example
action	The relabeling action to perform. For `labelmap`, it maps labels based on a regex.	`labelmap`
source_labels	Labels to match and extract values from.	`[__name__]`
regex	Regular expression to capture parts of the source label value.	`(.*)`
replacement	Replacement string using captured groups from `regex`. Use `$1`, `$2`, etc.	`metric_$1`
target_label	The label name to write the result to.	`new_metric_name`

Example: Using labelmap to rename metrics

relabel_configs:
  - source_labels: [__name__]
    regex: (.*)
    replacement: metric_$1
    target_label: new_metric_name
    action: labelmap

9. Implementing labeldrop Action

Action	Description	Usage	Example
labeldrop	Drops labels from metrics that match a specified regex pattern.	Used in relabeling configurations to remove unwanted labels.	`action: labeldrop regex: "^__.*__"`

Example: Dropping internal labels

relabel_configs:
  - action: labeldrop
    regex: "^__.*__"

Note: The labeldrop action removes labels matching the provided regex. It is commonly used to remove internal Prometheus labels like __name__ or __address__.

10. Using labelkeep Action

Property	Description	Example
action	The relabeling action to apply. For `labelkeep`, only metrics with the specified label are kept.	`labelkeep`
source_labels	List of label names to use as input for the relabeling operation.	`[__name__, instance]`
regex	Regular expression to match label values. Only labels matching this regex are kept.	`^job_.*`
replacement	Replacement string for matched labels. If not specified, the original label value is used.	`$1`
target_label	Label name to write the result to. If not specified, the original label is modified.	`new_label`

Example: Using labelkeep to filter metrics by job name

scrape_configs:
  - job_name: 'prometheus'
    relabel_configs:
      - action: labelkeep
        source_labels: [job]
        regex: '^prometheus.*'

Note: The labelkeep action is useful for filtering metrics based on label values to reduce the number of time series stored in Prometheus.

11. Understanding hashmod Action

Property	Description	Values	Notes
action	The relabeling action to perform	`hashmod`	Modifies the label value using a hash function
source_labels	Labels to use as input for the hash function	Array of label names	Required field
target_label	Label to write the result to	String	If not specified, defaults to `__tmp_hashmod`
modulus	Modulo value to apply to the hash result	Positive integer	Required field
separator	Separator to use between label values	String	Defaults to `;`

Example: Using hashmod to distribute metrics

relabel_configs:
  - source_labels: [__name__, instance]
    target_label: job_instance_hash
    action: hashmod
    modulus: 100

Note: The hashmod action is useful for distributing metrics across multiple targets or shards in a consistent way.

12. Configuring Metric Relabeling

Field	Description	Example
action	The relabeling action to apply. Valid values: `replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`.	`replace`
source_labels	List of label names to read values from.	`[__name__, instance]`
separator	Separator used to join source label values. Default is `;`.	`,`
target_label	Label to write the result to. Required for `replace`, `hashmod`, and `labelmap`.	`job`
regex	Regular expression to match source label values. Default is `(.*)`.	`^.*_([0-9]+)$`
replacement	Replacement string for `replace` action. Supports regex capture groups.	`$1`
modulus	Modulus used for `hashmod` action. Must be a positive integer.	`10`
labelmap	Regex pattern to match label names. Used with `labelmap` action.	`^__meta_(.*)$`

Example: Replace label value using regex

relabel_configs:
  - source_labels: [__name__]
    regex: (.*_)(\d+)
    target_label: job
    replacement: $1

Example: Drop metrics based on label value

relabel_configs:
  - source_labels: [job]
    regex: (.*_test)
    action: drop

Example: Hashmod for load balancing

relabel_configs:
  - source_labels: [instance]
    target_label: __tmp_hash
    action: hashmod
    modulus: 100

Note: Relabeling rules are processed in order, and each rule can modify labels for subsequent rules. The __tmp_ prefix is often used for temporary labels to avoid conflicts.