Working with Labels and Metadata

1. Creating Metric Labels

Label Name	Value Format	Description
job	`string`	Identifies the job that is scraping the metric
instance	`string`	Identifies the instance that is scraping the metric
status	`string`	Represents the status of a service or component (e.g., `up`, `down`)
environment	`string`	Environment where the metric is collected (e.g., `production`, `staging`)
version	`string`	Version of the application or service being monitored

Example: Adding Labels to a Metric

const client = require('prom-client');

// Create a counter with labels
const httpRequestCounter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'status']
});

// Increment the counter with labels
httpRequestCounter.inc({ method: 'GET', status: '200' });

2. Understanding Label Naming Conventions

Rule	Description	Examples
Label Name Format	Must match the regex `^[a-zA-Z_][a-zA-Z0-9_]*$`	`job`, `instance`, `status_code`
Reserved Prefixes	Labels starting with `__` are reserved for Prometheus internal use	`__name__`, `__error__`
Case Sensitivity	Label names are case-sensitive	`job` ≠ `Job`
Allowed Characters	Alphanumeric characters and underscores only	`http_status_code`, `node_id`
Length Limit	Label names must not exceed 128 characters	`very_long_label_name_that_exceeds_limit` (invalid)

Note: Label names should be descriptive and consistent across metrics to ensure clarity in queries and dashboards.

Example: Valid Label Names

http_requests_total{method="GET", status_code="200", instance="localhost:8080"}

3. Using Target Labels

Label Name	Description	Example	Notes
job	Identifies the job that scraped the metric	`job="prometheus"`	Required for all targets
instance	Uniquely identifies a target instance	`instance="localhost:9090"`	Usually derived from the target's address
environment	Environment where the target is running (e.g., dev, staging, prod)	`environment="production"`	Used for filtering and grouping
service	Service name the target belongs to	`service="api-server"`	Helps in service-level monitoring
version	Version of the application or service	`version="v2.30.0"`	Useful for tracking upgrades

Example: Adding Target Labels in Prometheus Configuration

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: [ "localhost:9090" ]
        labels:
          environment: "production"
          service: "prometheus"
          version: "v2.30.0"

Note: Label names must match the regex [a-zA-Z_][a-zA-Z0-9_]*. Label values can contain any Unicode characters.

Warning: Avoid high-cardinality labels (e.g., user ID, session ID) as they can lead to performance issues and increased memory usage.

4. Understanding Instrumentation Labels

Label Name	Description	Usage	Examples
job	Identifies the job that is being scraped	Required for all metrics	`job="prometheus"`
instance	Identifies the instance being scraped	Unique per job	`instance="localhost:9090"`
status	Represents the status of a service or operation	Used for operational metrics	`status="up"`
method	HTTP method used in a request	For HTTP metrics	`method="GET"`
path	HTTP path being accessed	For HTTP metrics	`path="/api/v1/query"`
error	Indicates an error occurred in a request	For error tracking	`error="timeout"`

Example: Instrumenting a Counter with Labels

const client = require('prom-client');

// Create a counter with labels
const httpRequestCounter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'path', 'status']
});

// Increment the counter with labels
httpRequestCounter.inc({ method: 'GET', path: '/api/v1/query', status: '200' });

Note: Label values should be kept low cardinality to avoid performance issues.

Warning: Adding too many labels or high cardinality labels can lead to memory issues and slow query performance.

5. Using Meta Labels

Label Name	Description	Usage	Notes
__name__	Identifier for the metric	Used in queries to select metrics by name	Read-only, automatically set by Prometheus
__meta_job__	Job name from configuration	Used to group metrics by job	Set in Prometheus configuration files
__meta_instance__	Instance identifier	Used to distinguish between instances of the same job	Typically set by service discovery mechanisms
__meta_scheme__	Protocol scheme (http or https)	Used in scraping configurations	Set by service discovery or static config
__meta_path__	Path to scrape from	Used in scraping configurations	Set by service discovery or static config

Example: Querying with Meta Labels

up{__meta_job__="web-server"} == 1

Example: Filtering by Instance

http_requests_total{__meta_instance__="web-server-01:8080"}

Note: Meta labels are automatically added by Prometheus during service discovery and can be used in recording rules, alerts, and queries.

6. Handling High-Cardinality Labels

Label Name	Description	Impact	Best Practice
instance	Identifies the target instance	High cardinality if many instances exist	Use consistent naming and limit instance count
job	Identifies the job that scrapes a target	Medium cardinality, often reused	Use descriptive job names to avoid conflicts
endpoint	Identifies the endpoint being scraped	High if many endpoints per instance	Avoid dynamic endpoint labels; use static ones

Example: Label Cardinality Issue

metrics:
  - name: http_requests_total
    help: Total number of HTTP requests
    type: counter
    labels:
      instance: "host123"
      job: "webapp"
      endpoint: "/api/v1/users/123456789"

Note: High cardinality labels can lead to increased memory usage and reduced query performance in Prometheus.

Warning: Avoid generating labels with unique values for each metric sample (e.g., user IDs, session IDs).

Strategies to Reduce Label Cardinality

Use job and instance consistently across targets
Avoid adding labels with many unique values
Use label relabeling rules to drop or remap labels
Monitor Prometheus metrics for high-cardinality label usage

7. Implementing Label Inheritance

Label inheritance in Prometheus allows metrics to inherit labels from higher-level scopes, reducing redundancy and improving consistency across time series.

Label Type	Description	Example	Inheritance Context
job	Identifies the job that is being scraped	`job="node_exporter"`	Inherited from scrape configuration
instance	Identifies the instance being scraped	`instance="localhost:9100"`	Inherited from scrape configuration
environment	Environment label for grouping	`environment="production"`	Manually added to metrics

Example: Adding Inherited Labels to Metrics

groups:
- name: node
  rules:
  - alert: HighCpuUsage
    expr: node_cpu_seconds_total{mode="idle"} < 0.1
    labels:
      severity: page
      environment: production

Note: Prometheus does not support true label inheritance like some other systems. Labels must be explicitly defined or added via recording rules.

Label Inheritance Best Practices

Use consistent label names across services and jobs
Define common labels in scrape configs or recording rules
Avoid over-labeling which can increase cardinality

8. Using Label Matchers

Matcher Type	Description	Examples	Notes
=	Exact match	`job="prometheus"`	Matches labels with exact value
!=	Not equal	`job!="prometheus"`	Excludes labels with specified value
=~	Regex match	`job=~"prom.*"`	Matches labels matching regex pattern
!~	Regex not match	`job!~"prom.*"`	Excludes labels matching regex pattern

Example: Using Label Matchers in Queries

up{job=~"prometheus|alertmanager"} == 1

Example: Excluding Specific Jobs

http_requests_total{job!~"test.*"}

Note: Label matchers are used in PromQL queries to filter time series by label values. They can be combined with logical operators like and, or, and unless.

9. Understanding Label Precedence

Label Source	Priority	Description
Metric Name	`1`	Primary identifier; highest precedence
Static Labels	`2`	Labels added at scrape time via configuration
Dynamic Labels	`3`	Labels from metric exposition or relabeling rules
Relabeled Labels	`4`	Labels modified by relabeling rules (lowest priority)

Note: When multiple labels share the same name, Prometheus uses the label with the highest priority to determine the final metric value.

Example: Label Precedence in Action

# Configuration file (prometheus.yml)
scrape_configs:
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['localhost:9100']
        labels:
          env: 'production'
          region: 'us-west-1'

    relabel_configs:
      - source_labels: [env]
        target_label: env
        replacement: 'staging'

Example: Metric with Label Precedence

# Sample metric output
node_cpu_seconds_total{env='staging',region='us-west-1',job='node_exporter'} 1234.56

# Explanation:
# - env='staging' comes from relabeling (priority 4)
# - region='us-west-1' comes from static config (priority 2)
# - job='node_exporter' is default from scrape config (priority 1)

10. Dropping Unnecessary Labels

Label Name	Description	Action
job	Identifies the job that scraped the metric	DEPRECATED in favor of `instance`
instance	Identifies the instance that scraped the metric	Keep for detailed monitoring
__name__	Represents the metric name itself	Do not drop — essential for querying
severity	Used for alerting severity levels	Keep if used in alerting rules
environment	Environment identifier (e.g., dev, staging, prod)	Keep for filtering and grouping

Example: Dropping labels via relabeling

scrape_configs:
  - job_name: 'myapp'
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_probe]
        target_label: __tmp_probe
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        target_label: __metrics_path__
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __tmp_port
      - action: drop
        source_labels: [__tmp_probe]
        regex: '^false

Note: Use relabel_configs to drop labels that are not needed for monitoring or alerting. This reduces metric cardinality and improves performance.

11. Adding Static Labels

Label Name	Value	Description
job	`prometheus`	Identifies the job that scraped the metric
instance	`localhost:9090`	Identifies the instance that scraped the metric
environment	`production`	Environment label for grouping
version	`2.37.0`	Version of the Prometheus server

Example: Adding static labels via configuration

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
        labels:
          environment: production
          version: 2.37.0

12. Using External Labels

Label Name	Description	Usage	Example
job	Identifies the job that is scraping the target	Used to group metrics by job name	`job="prometheus"`
instance	Identifies the instance that is being scraped	Uniquely identifies a target within a job	`instance="localhost:9090"`
environment	Environment where the target is running (e.g., dev, staging, prod)	Used for filtering and grouping metrics by environment	`environment="production"`
region	Geographic region where the target is located	Used for regional analytics and monitoring	`region="us-west-1"`

Example: Configuring External Labels in Prometheus

global:
  external_labels:
    environment: production
    region: us-west-1
    job: prometheus

Note: External labels are applied to all metrics scraped by the Prometheus instance and cannot be overridden by metric labels.