Understanding Prometheus Metric Types
1. Working with Counter Metrics
| Property | Description | Example |
|---|---|---|
| counter_name | Unique identifier for the counter metric | http_requests_total |
| counter_help | Help text describing the metric | Total number of HTTP requests |
| counter_value | Current value of the counter (automatically incremented) | 125 |
Example: Incrementing a Counter
const client = require('prom-client');
const httpRequests = new client.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'code']
});
// Increment the counter
httpRequests.inc({ method: 'GET', code: '200' });
Example: Counter with Labels
const client = require('prom-client');
const requests = new client.Counter({
name: 'requests_total',
help: 'Total number of requests by method and status',
labelNames: ['method', 'status']
});
// Increment with labels
requests.inc({ method: 'POST', status: 'success' });
requests.inc({ method: 'GET', status: 'error' });
Note: Counters are monotonically increasing and reset on restart.
2. Working with Gauge Metrics
| Property | Description | Example |
|---|---|---|
| gauge_name | Unique identifier for the gauge metric | http_requests_total |
| gauge_help | Help text describing the metric | Total number of HTTP requests |
| gauge_value | Current value of the gauge | 125 |
Example: Setting a Gauge Value
const client = require('prom-client');
const gauge = new client.Gauge({
name: 'nodejs_heap_used_bytes',
help: 'The number of bytes of the Node.js heap used'
});
gauge.set(1024 * 1024); // Set to 1MB
Example: Incrementing a Gauge
const client = require('prom-client');
const gauge = new client.Gauge({
name: 'active_users',
help: 'Number of active users'
});
gauge.inc(5); // Increment by 5
Example: Decrementing a Gauge
const client = require('prom-client');
const gauge = new client.Gauge({
name: 'active_users',
help: 'Number of active users'
});
gauge.dec(2); // Decrement by 2
Note: Gauge metrics can be set to any numeric value, including negative values. They are useful for tracking metrics like memory usage or active user counts.
3. Working with Histogram Metrics
| Metric Name | Labels | Description | Example |
|---|---|---|---|
| http_request_duration_seconds | method, path, status |
Duration of HTTP requests in seconds | http_request_duration_seconds{method="GET",path="/api/users",status="200"} 0.05 |
| process_start_time_seconds | pid |
Start time of the process in seconds since Unix epoch | process_start_time_seconds{pid="12345"} 1609459200.0 |
Example: Histogram Configuration
groups:
- name: http
rules:
- alert: HighRequestLatency
expr: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
for: 10m
labels:
severity: page
annotations:
summary: "High request latency"
Example: Querying Histogram Quantiles
histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le))
Note: The
le label in histogram buckets represents the upper bound of each bucket.
Warning: Avoid using too many buckets as it can lead to high cardinality and performance issues.
Key Points About Histograms
- Histograms are used to track the distribution of values over time.
- They use buckets to count observations in different ranges.
- The
le(less than or equal to) label defines the bucket boundaries. - Quantiles can be calculated using
histogram_quantile.
4. Working with Summary Metrics
| Metric Type | Description | Key Characteristics | Usage Example |
|---|---|---|---|
| summary | Tracks the distribution of observations over time | Can compute quantiles, includes count and sum | http_request_duration_seconds_summary{quantile="0.95"} 123.45 |
Example: Creating a Summary Metric
const client = require('prom-client');
const httpRequestDuration = new client.Summary({
name: 'http_request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
percentiles: [0.5, 0.9, 0.99],
});
// Observing a request duration
httpRequestDuration.observe(123.45);
Note: Summary metrics are useful for tracking distributions of values, such as response times or request sizes. They provide quantiles (e.g., 50th, 90th, 99th percentiles) to understand the distribution of values.
Summary Metric Properties
name: Unique identifier for the metrichelp: Description of what the metric measurespercentiles: Array of quantiles to calculate (e.g., [0.5, 0.9, 0.99])maxAge: Maximum age of observations to keep in memory (default: 600 seconds)ageBuckets: Number of buckets to use for age-based decay (default: 1)
5. Understanding Metric Naming Conventions
| Convention | Description | Examples |
|---|---|---|
| metric_name | Must be a valid ASCII identifier, lowercase letters and digits only | http_requests_total |
| metric_name with underscores | Underscores are allowed, but not hyphens | node_cpu_seconds_total |
| metric_name with suffixes | Use standard suffixes to indicate metric type | http_requests_total, node_cpu_seconds_total |
| metric_name with units | Include unit names in metric names when appropriate | http_request_duration_seconds |
| metric_name with labels | Labels provide additional dimensions to metrics | http_requests_total{method="GET",status="200"} |
Example: Naming Convention Best Practices
http_requests_total
node_cpu_seconds_total
process_start_time_seconds
Note: Avoid using special characters or spaces in metric names. Stick to lowercase letters, digits, and underscores.
Warning: Metric names should be descriptive but concise. Avoid overly long names that reduce readability.
6. Using Metric Labels
| Label Name | Description | Example Value | Usage |
|---|---|---|---|
| job | Identifies the job that is scraping the metric | node_exporter |
Required for most Prometheus setups |
| instance | Identifies the specific instance being scraped | localhost:9100 |
Used to differentiate between multiple instances of same job |
| environment | Environment where the metric originates (e.g., dev, staging, prod) | production |
Used for filtering and grouping metrics by environment |
| service | Name of the service that exposes this metric | api-server |
Helps in organizing and filtering metrics by service |
| status | State or status of the service (e.g., up, down) | up |
Used in alerting and monitoring system health |
Example: Adding Labels to a Counter Metric
const httpRequestCounter = new client.Counter({
name: "http_requests_total",
help: "Total number of HTTP requests",
labelNames: ["method", "status"]
});
httpRequestCounter.inc({ method: "GET", status: "200" });
Note: Label names must match the regex
[a-zA-Z_:][a-zA-Z0-9_:]*. Label values can contain any Unicode characters.
Warning: Avoid high-cardinality labels (e.g., user ID, session ID) as they can lead to performance issues and high memory usage in Prometheus.
7. Understanding Reserved Labels
| Label Name | Description | Usage | Notes |
|---|---|---|---|
| __name__ | Internal metric name | Used for grouping and filtering | Automatically added by Prometheus |
| job | Job name that scraped the metric | Used for identifying scrape jobs | Set in Prometheus configuration or via relabeling |
| instance | Instance identifier | Used for distinguishing between instances | Set in Prometheus configuration or via relabeling |
| scrape_time | Time when the metric was scraped | Used for debugging and monitoring scrape behavior | Added automatically by Prometheus |
| scrape_duration_seconds | Duration of the scrape operation | Used for performance monitoring | Added automatically by Prometheus |
Note: Reserved labels are automatically added by Prometheus and should not be overridden in user-defined metrics.
8. Understanding Metric Exposition Format
| Metric Type | Format | Description |
|---|---|---|
| Counter | metric_name_total{label="value"} 123 |
Monotonically increasing counter, always starts at 0 |
| Gauge | metric_name{label="value"} 456.7 |
Arbitrary float value that can go up or down |
| Histogram | metric_name_bucket{le="0.5",label="value"} 10metric_name_sum{label="value"} 123.4metric_name_count{label="value"} 50 |
Tracks observations and calculates quantiles |
| Summary | metric_name{quantile="0.9",label="value"} 12.3metric_name_sum{label="value"} 123.4metric_name_count{label="value"} 50 |
Tracks observations and calculates quantiles (server-side aggregation) |
| Untyped | metric_name{label="value"} 123 |
Used for unknown metric types or when type is not specified |
Example: Counter exposition
# TYPE http_requests_total counter
http_requests_total{method="get",code="200"} 123
http_requests_total{method="post",code="200"} 45
Example: Histogram exposition
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.05"} 10
request_duration_seconds_bucket{le="0.1"} 25
request_duration_seconds_bucket{le="0.5"} 40
request_duration_seconds_bucket{le="1"} 45
request_duration_seconds_bucket{le="+Inf"} 50
request_duration_seconds_sum 12.3
request_duration_seconds_count 50
Note: The exposition format is a text-based protocol that Prometheus uses to scrape metrics. Each metric line includes the metric name, labels (in curly braces), and value.
9. Working with Timestamps
| Metric Type | Timestamp Handling | Notes |
|---|---|---|
| Counter | timestamp is optional; if omitted, current time is used |
Use timestamp to set a custom time for the metric value |
| Gauge | timestamp is optional; if omitted, current time is used |
Use timestamp to set a custom time for the metric value |
| Histogram | timestamp is optional; if omitted, current time is used |
Use timestamp to set a custom time for the metric value |
| Summary | timestamp is optional; if omitted, current time is used |
Use timestamp to set a custom time for the metric value |
| Untyped | timestamp is optional; if omitted, current time is used |
Use timestamp to set a custom time for the metric value |
Example: Setting Custom Timestamps
const client = new PrometheusClient();
// Counter with custom timestamp
client.registerCounter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'code'],
values: {
method: 'GET',
code: '200'
},
timestamp: new Date('2023-10-01T12:00:00Z')
});
// Gauge with custom timestamp
client.registerGauge({
name: 'memory_usage_bytes',
help: 'Memory usage in bytes',
value: 1024 * 1024,
timestamp: new Date('2023-10-01T12:05:00Z')
});
Note: Timestamps in Prometheus are in milliseconds since Unix epoch (1970-01-01 00:00:00 UTC). Ensure that timestamps are not in the future or too far in the past, as they may be ignored by Prometheus.
10. Using Metric Help Text
Metric help text provides human-readable descriptions for Prometheus metrics, enhancing observability and understanding.
| Field | Description | Example |
|---|---|---|
| help_text | Descriptive text explaining the metric's purpose and usage | Number of HTTP requests handled by the server |
Example: Defining Help Text in a Metric
# HELP http_requests_total Total number of HTTP requests handled by the server
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1234
Note: Help text is optional but highly recommended for clarity and maintainability.
11. Choosing Histogram vs Summary
| Aspect | Histogram | Summary |
|---|---|---|
| Data Type | Count of observations in buckets | Quantiles (percentiles) |
| Calculation | Pre-calculated buckets | Calculated at query time |
| Storage | Multiple time series (one per bucket) | Single time series |
| Query Performance | Fast, efficient | Slower, requires more resources |
| Use Case | High cardinality, detailed distribution | Low cardinality, specific percentiles |
| Configurable Buckets | Yes, custom buckets | No, fixed quantiles |
| Percentile Accuracy | Approximate | Exact (within tolerance) |
Example: Histogram Usage
http_request_duration_seconds_bucket{le="0.05",method="GET"} 123
http_request_duration_seconds_bucket{le="0.1",method="GET"} 234
http_request_duration_seconds_bucket{le="0.5",method="GET"} 345
http_request_duration_seconds_count{method="GET"} 456
http_request_duration_seconds_sum{method="GET"} 123.45
Example: Summary Usage
http_request_duration_seconds{quantile="0.5",method="GET"} 0.03
http_request_duration_seconds{quantile="0.9",method="GET"} 0.07
http_request_duration_seconds{quantile="0.99",method="GET"} 0.12
http_request_duration_seconds_count{method="GET"} 456
http_request_duration_seconds_sum{method="GET"} 123.45
Decision Guide
- Use Histogram when you need to calculate percentiles across multiple time series or have high cardinality data.
- Use Summary when you only need a few specific percentiles and want exact values.
- Histograms are more efficient for large-scale systems with many metrics.
- Summaries provide better accuracy for specific quantiles but use more memory and CPU at query time.
12. Understanding Counter Resets
| Metric Type | Description | Reset Behavior | Notes |
|---|---|---|---|
| counter | Tracks cumulative values that only increase | DEPRECATED in v2.0+ — counters do not reset automatically | Counter values are monotonically increasing and reset only on restart or metric removal |
| gauge | Represents a single numerical value that can go up and down | No automatic reset | Gauge values are not reset unless explicitly set to a new value or removed |
| histogram | Tracks the distribution of values over time | DEPRECATED in v2.0+ — histogram resets are handled by buckets | Buckets reset on metric removal or restart, not on value updates |
| summary | Tracks the distribution of values over time | DEPRECATED in v2.0+ — summary resets are handled by quantiles | Quantiles reset on metric removal or restart, not on value updates |
Note: In Prometheus v2.0+, counter resets are no longer automatic, and the
reset behavior is deprecated. Counters must be manually reset or removed to simulate a reset.Example: Counter Reset Simulation
// Simulate counter reset using a gauge to track the last value
const client = require('prom-client');
// Create a counter
const counter = new client.Counter({
name: 'requests_total',
help: 'Total number of requests',
});
// Simulate reset logic using a gauge
const lastValue = new client.Gauge({
name: 'last_request_value',
help: 'Last known counter value before reset'
});
// Function to simulate a reset
function simulateReset() {
const currentValue = counter.get();
lastValue.set(currentValue);
counter.reset(); // Reset the counter to zero
}
// Example usage:
counter.inc();
simulateReset();
// The gauge now holds the previous value for reference
console.log('Last counter value:', lastValue.get());