Skip to main content

Understanding Prometheus Metric Types

1. Working with Counter Metrics

Property Description Example
counter_name Unique identifier for the counter metric http_requests_total
counter_help Help text describing the metric Total number of HTTP requests
counter_value Current value of the counter (automatically incremented) 125

Example: Incrementing a Counter

const client = require('prom-client');

const httpRequests = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'code']
});

// Increment the counter
httpRequests.inc({ method: 'GET', code: '200' });

Example: Counter with Labels

const client = require('prom-client');

const requests = new client.Counter({
  name: 'requests_total',
  help: 'Total number of requests by method and status',
  labelNames: ['method', 'status']
});

// Increment with labels
requests.inc({ method: 'POST', status: 'success' });
requests.inc({ method: 'GET', status: 'error' });
Note: Counters are monotonically increasing and reset on restart.

2. Working with Gauge Metrics

Property Description Example
gauge_name Unique identifier for the gauge metric http_requests_total
gauge_help Help text describing the metric Total number of HTTP requests
gauge_value Current value of the gauge 125

Example: Setting a Gauge Value

const client = require('prom-client');

const gauge = new client.Gauge({
  name: 'nodejs_heap_used_bytes',
  help: 'The number of bytes of the Node.js heap used'
});

gauge.set(1024 * 1024); // Set to 1MB

Example: Incrementing a Gauge

const client = require('prom-client');

const gauge = new client.Gauge({
  name: 'active_users',
  help: 'Number of active users'
});

gauge.inc(5); // Increment by 5

Example: Decrementing a Gauge

const client = require('prom-client');

const gauge = new client.Gauge({
  name: 'active_users',
  help: 'Number of active users'
});

gauge.dec(2); // Decrement by 2
Note: Gauge metrics can be set to any numeric value, including negative values. They are useful for tracking metrics like memory usage or active user counts.

3. Working with Histogram Metrics

Metric Name Labels Description Example
http_request_duration_seconds method, path, status Duration of HTTP requests in seconds http_request_duration_seconds{method="GET",path="/api/users",status="200"} 0.05
process_start_time_seconds pid Start time of the process in seconds since Unix epoch process_start_time_seconds{pid="12345"} 1609459200.0

Example: Histogram Configuration

groups:
- name: http
  rules:
  - alert: HighRequestLatency
    expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High request latency"

Example: Querying Histogram Quantiles

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
Note: The le label in histogram buckets represents the upper bound of each bucket.
Warning: Avoid using too many buckets as it can lead to high cardinality and performance issues.

Key Points About Histograms

  • Histograms are used to track the distribution of values over time.
  • They use buckets to count observations in different ranges.
  • The le (less than or equal to) label defines the bucket boundaries.
  • Quantiles can be calculated using histogram_quantile.

4. Working with Summary Metrics

Metric Type Description Key Characteristics Usage Example
summary Tracks the distribution of observations over time Can compute quantiles, includes count and sum http_request_duration_seconds_summary{quantile="0.95"} 123.45

Example: Creating a Summary Metric

const client = require('prom-client');

const httpRequestDuration = new client.Summary({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  percentiles: [0.5, 0.9, 0.99],
});

// Observing a request duration
httpRequestDuration.observe(123.45);
Note: Summary metrics are useful for tracking distributions of values, such as response times or request sizes. They provide quantiles (e.g., 50th, 90th, 99th percentiles) to understand the distribution of values.

Summary Metric Properties

  • name: Unique identifier for the metric
  • help: Description of what the metric measures
  • percentiles: Array of quantiles to calculate (e.g., [0.5, 0.9, 0.99])
  • maxAge: Maximum age of observations to keep in memory (default: 600 seconds)
  • ageBuckets: Number of buckets to use for age-based decay (default: 1)

5. Understanding Metric Naming Conventions

Convention Description Examples
metric_name Must be a valid ASCII identifier, lowercase letters and digits only http_requests_total
metric_name with underscores Underscores are allowed, but not hyphens node_cpu_seconds_total
metric_name with suffixes Use standard suffixes to indicate metric type http_requests_total, node_cpu_seconds_total
metric_name with units Include unit names in metric names when appropriate http_request_duration_seconds
metric_name with labels Labels provide additional dimensions to metrics http_requests_total{method="GET",status="200"}

Example: Naming Convention Best Practices


http_requests_total
node_cpu_seconds_total
process_start_time_seconds
Note: Avoid using special characters or spaces in metric names. Stick to lowercase letters, digits, and underscores.
Warning: Metric names should be descriptive but concise. Avoid overly long names that reduce readability.

6. Using Metric Labels

Label Name Description Example Value Usage
job Identifies the job that is scraping the metric node_exporter Required for most Prometheus setups
instance Identifies the specific instance being scraped localhost:9100 Used to differentiate between multiple instances of same job
environment Environment where the metric originates (e.g., dev, staging, prod) production Used for filtering and grouping metrics by environment
service Name of the service that exposes this metric api-server Helps in organizing and filtering metrics by service
status State or status of the service (e.g., up, down) up Used in alerting and monitoring system health

Example: Label Usage in Prometheus Query

up{job="node_exporter", environment="production"}

Example: Adding Labels to a Counter Metric

const httpRequestCounter = new client.Counter({
  name: "http_requests_total",
  help: "Total number of HTTP requests",
  labelNames: ["method", "status"]
});

httpRequestCounter.inc({ method: "GET", status: "200" });
Note: Label names must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*. Label values can contain any Unicode characters.
Warning: Avoid high-cardinality labels (e.g., user ID, session ID) as they can lead to performance issues and high memory usage in Prometheus.

7. Understanding Reserved Labels

Label Name Description Usage Notes
__name__ Internal metric name Used for grouping and filtering Automatically added by Prometheus
job Job name that scraped the metric Used for identifying scrape jobs Set in Prometheus configuration or via relabeling
instance Instance identifier Used for distinguishing between instances Set in Prometheus configuration or via relabeling
scrape_time Time when the metric was scraped Used for debugging and monitoring scrape behavior Added automatically by Prometheus
scrape_duration_seconds Duration of the scrape operation Used for performance monitoring Added automatically by Prometheus
Note: Reserved labels are automatically added by Prometheus and should not be overridden in user-defined metrics.

8. Understanding Metric Exposition Format

Metric Type Format Description
Counter metric_name_total{label="value"} 123 Monotonically increasing counter, always starts at 0
Gauge metric_name{label="value"} 456.7 Arbitrary float value that can go up or down
Histogram metric_name_bucket{le="0.5",label="value"} 10
metric_name_sum{label="value"} 123.4
metric_name_count{label="value"} 50
Tracks observations and calculates quantiles
Summary metric_name{quantile="0.9",label="value"} 12.3
metric_name_sum{label="value"} 123.4
metric_name_count{label="value"} 50
Tracks observations and calculates quantiles (server-side aggregation)
Untyped metric_name{label="value"} 123 Used for unknown metric types or when type is not specified

Example: Counter exposition

# TYPE http_requests_total counter
http_requests_total{method="get",code="200"} 123
http_requests_total{method="post",code="200"} 45

Example: Histogram exposition

# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.05"} 10
request_duration_seconds_bucket{le="0.1"} 25
request_duration_seconds_bucket{le="0.5"} 40
request_duration_seconds_bucket{le="1"} 45
request_duration_seconds_bucket{le="+Inf"} 50
request_duration_seconds_sum 12.3
request_duration_seconds_count 50
Note: The exposition format is a text-based protocol that Prometheus uses to scrape metrics. Each metric line includes the metric name, labels (in curly braces), and value.

9. Working with Timestamps

Metric Type Timestamp Handling Notes
Counter timestamp is optional; if omitted, current time is used Use timestamp to set a custom time for the metric value
Gauge timestamp is optional; if omitted, current time is used Use timestamp to set a custom time for the metric value
Histogram timestamp is optional; if omitted, current time is used Use timestamp to set a custom time for the metric value
Summary timestamp is optional; if omitted, current time is used Use timestamp to set a custom time for the metric value
Untyped timestamp is optional; if omitted, current time is used Use timestamp to set a custom time for the metric value

Example: Setting Custom Timestamps

const client = new PrometheusClient();

// Counter with custom timestamp
client.registerCounter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'code'],
  values: {
    method: 'GET',
    code: '200'
  },
  timestamp: new Date('2023-10-01T12:00:00Z')
});

// Gauge with custom timestamp
client.registerGauge({
  name: 'memory_usage_bytes',
  help: 'Memory usage in bytes',
  value: 1024 * 1024,
  timestamp: new Date('2023-10-01T12:05:00Z')
});
Note: Timestamps in Prometheus are in milliseconds since Unix epoch (1970-01-01 00:00:00 UTC). Ensure that timestamps are not in the future or too far in the past, as they may be ignored by Prometheus.

10. Using Metric Help Text

Metric help text provides human-readable descriptions for Prometheus metrics, enhancing observability and understanding.

Field Description Example
help_text Descriptive text explaining the metric's purpose and usage Number of HTTP requests handled by the server

Example: Defining Help Text in a Metric

# HELP http_requests_total Total number of HTTP requests handled by the server
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1234
Note: Help text is optional but highly recommended for clarity and maintainability.

11. Choosing Histogram vs Summary

Aspect Histogram Summary
Data Type Count of observations in buckets Quantiles (percentiles)
Calculation Pre-calculated buckets Calculated at query time
Storage Multiple time series (one per bucket) Single time series
Query Performance Fast, efficient Slower, requires more resources
Use Case High cardinality, detailed distribution Low cardinality, specific percentiles
Configurable Buckets Yes, custom buckets No, fixed quantiles
Percentile Accuracy Approximate Exact (within tolerance)

Example: Histogram Usage

http_request_duration_seconds_bucket{le="0.05",method="GET"} 123
http_request_duration_seconds_bucket{le="0.1",method="GET"} 234
http_request_duration_seconds_bucket{le="0.5",method="GET"} 345
http_request_duration_seconds_count{method="GET"} 456
http_request_duration_seconds_sum{method="GET"} 123.45

Example: Summary Usage

http_request_duration_seconds{quantile="0.5",method="GET"} 0.03
http_request_duration_seconds{quantile="0.9",method="GET"} 0.07
http_request_duration_seconds{quantile="0.99",method="GET"} 0.12
http_request_duration_seconds_count{method="GET"} 456
http_request_duration_seconds_sum{method="GET"} 123.45

Decision Guide

  • Use Histogram when you need to calculate percentiles across multiple time series or have high cardinality data.
  • Use Summary when you only need a few specific percentiles and want exact values.
  • Histograms are more efficient for large-scale systems with many metrics.
  • Summaries provide better accuracy for specific quantiles but use more memory and CPU at query time.

12. Understanding Counter Resets

Metric Type Description Reset Behavior Notes
counter Tracks cumulative values that only increase DEPRECATED in v2.0+ — counters do not reset automatically Counter values are monotonically increasing and reset only on restart or metric removal
gauge Represents a single numerical value that can go up and down No automatic reset Gauge values are not reset unless explicitly set to a new value or removed
histogram Tracks the distribution of values over time DEPRECATED in v2.0+ — histogram resets are handled by buckets Buckets reset on metric removal or restart, not on value updates
summary Tracks the distribution of values over time DEPRECATED in v2.0+ — summary resets are handled by quantiles Quantiles reset on metric removal or restart, not on value updates
Note: In Prometheus v2.0+, counter resets are no longer automatic, and the reset behavior is deprecated. Counters must be manually reset or removed to simulate a reset.

Example: Counter Reset Simulation

// Simulate counter reset using a gauge to track the last value
const client = require('prom-client');

// Create a counter
const counter = new client.Counter({
  name: 'requests_total',
  help: 'Total number of requests',
});

// Simulate reset logic using a gauge
const lastValue = new client.Gauge({
  name: 'last_request_value',
  help: 'Last known counter value before reset'
});

// Function to simulate a reset
function simulateReset() {
  const currentValue = counter.get();
  lastValue.set(currentValue);
  counter.reset(); // Reset the counter to zero
}

// Example usage:
counter.inc();
simulateReset();

// The gauge now holds the previous value for reference
console.log('Last counter value:', lastValue.get());

Was this article helpful?