Skip to main content

Implementing Custom Metrics

1. Using Client Libraries

Prometheus provides official client libraries for various programming languages to expose metrics from applications. These libraries handle the exposition of metrics in the Prometheus text format and provide convenient APIs for instrumenting code.

Language Library Repository Status
Go github.com/prometheus/client_golang GitHub NEW v1.19.0
Java io.prometheus.client GitHub NEW v0.16.0
Python prometheus-client GitHub NEW v0.19.0
JavaScript prom-client GitHub NEW v15.0.0
Ruby prometheus-client GitHub DEPRECATED v0.12.0

Example: Go Client Library

package main

import (
    "log"
    "net/http"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    httpRequestCount = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "http_requests_total",
            Help: "Total number of HTTP requests",
        },
        []string{"method", "code"},
    )
)

func main() {
    http.Handle("/metrics", promhttp.Handler())
    http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
        httpRequestCount.WithLabelValues(r.Method, "200").Inc()
        w.Write([]byte("Hello, World!"))
    })
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Example: Python Client Library

from prometheus_client import Counter, start_http_server

# Create a counter metric
http_requests_total = Counter(
    "http_requests_total",
    "Total number of HTTP requests",
    [ "method", "code" ]
)

def handle_request():
    # Increment the counter
    http_requests_total.labels(method="GET", code="200").inc()
    return "Hello, World!"

if __name__ == "__main__":
    start_http_server(8080)
Note: Client libraries automatically expose metrics at the /metrics endpoint by default, which Prometheus scrapes.

2. Creating Counter Metrics

Metric Name Type Description Usage Example
http_requests_total Counter Total number of HTTP requests handled by the application http_requests_total{method="GET",endpoint="/api/users"} 120
error_count Counter Number of errors encountered in the system error_count{type="database",code="500"} 5

Example: Registering a Counter in Go

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    httpRequestCount = promauto.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests",
    }, []string{"method", "endpoint"})
)

func main() {
    // Increment counter on each request
    httpRequestCount.WithLabelValues("GET", "/api/users").Inc()
}
Note: Counters are monotonically increasing and reset on restart. They are ideal for tracking cumulative values like total requests or errors.

3. Creating Gauge Metrics

Property Description Example
gauge.name Name of the gauge metric http_requests_total
gauge.help Help text describing the metric Total number of HTTP requests
gauge.value Current value of the gauge 120

Example: Creating a Gauge Metric

const client = require('prom-client');

const gauge = new client.Gauge({
  name: 'nodejs_heap_used_bytes',
  help: 'The number of bytes of the Node.js heap used',
});

// Set gauge value
gauge.set(process.memoryUsage().heapUsed);

// Increment gauge value
gauge.inc(1024); // Adds 1KB to the gauge

// Decrement gauge value
gauge.dec(512); // Removes 512 bytes from the gauge
Note: Gauges are used for metrics that can go up and down, like memory usage or queue sizes.

4. Creating Histogram Metrics

Metric Name Labels Description Usage
request_duration_seconds method, endpoint Duration of HTTP requests in seconds Used to monitor latency of API endpoints
response_size_bytes status_code Size of HTTP responses in bytes Tracks payload sizes for performance analysis

Example: Histogram Metric Definition

const promClient = require('prom-client');

const histogram = new promClient.Histogram({
  name: 'request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'endpoint'],
  buckets: [0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]
});

module.exports = histogram;

Example: Recording Histogram Values

const start = Date.now();
// Simulate request processing
await processRequest();

const duration = (Date.now() - start) / 1000;
histogram.observe({ method: 'GET', endpoint: '/api/users' }, duration);
Note: Histograms are ideal for measuring distributions of values like request durations or response sizes. They provide quantiles (e.g., 90th percentile) which help understand performance characteristics across different load conditions.

5. Creating Summary Metrics

Summary metrics in Prometheus are used to track the distribution of events, such as request durations or response sizes. They allow you to calculate quantiles over a sliding time window.

Property Description Default Value Notes
name The name of the summary metric None Metric names must follow Prometheus naming conventions
help A help text describing the metric None Help text is used in Prometheus UI and documentation
label_names List of label names to be included in the metric None Labels allow for grouping and filtering of metrics
max_age Maximum age of observations to keep in memory 60 seconds Controls how long observations are retained before being discarded
buf_cap Buffer capacity for observations 1000 Defines how many observations can be stored in the buffer

Example: Creating a Summary Metric in Go

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    requestDuration = promauto.NewSummary(prometheus.SummaryOpts{
        Name: "http_request_duration_seconds",
        Help: "Duration of HTTP requests in seconds",
        LabelNames: []string{"method", "endpoint"},
        MaxAge: 60 * time.Second,
        BufCap: 1000,
    })
)

func main() {
    // Example usage
    requestDuration.WithLabelValues("GET", "/api/users").Observe(0.5)
}
Note: Summary metrics are particularly useful for tracking latency distributions. They maintain a sliding window of observations and calculate quantiles over that window.

6. Exposing Metrics Endpoint

Property Description Default Value Notes
endpoint URL path to expose metrics /metrics Can be customized to any valid path
port Port to bind the metrics server 9090 Must be an available port on the host
handler Function to handle metric requests defaultHandler Can be replaced with custom logic

Example: Basic Metrics Endpoint Setup

const client = require('prom-client');
const express = require('express');

const app = express();
const collectDefaultMetrics = client.collectDefaultMetrics;
const register = client.register;

// Collect default metrics
collectDefaultMetrics({ prefix: 'myapp_' });

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

app.listen(9090, () => {
  console.log('Metrics endpoint available at http://localhost:9090/metrics');
});
Note: The metrics endpoint must be accessible to Prometheus for scraping. Ensure the port is open and not blocked by firewalls or network policies.
Warning: Exposing metrics on a public endpoint without authentication can lead to information leakage. Consider securing the endpoint with basic auth or other access controls.

7. Using Metric Registry

Property Description Default Value Notes
registry Instance of the Prometheus registry to register metrics with null Must be an instance of Prometheus.Registry
name Name of the metric to register undefined Must be a valid Prometheus metric name
help Help text for the metric undefined Descriptive text for the metric
labels Array of label names to include in the metric [] Labels must be strings

Example: Registering a Counter Metric

const Prometheus = require('prom-client');
const registry = new Prometheus.Registry();

const counter = new Prometheus.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'status']
});

registry.registerMetric(counter);

// Increment the counter
counter.inc({ method: 'GET', status: '200' });

Example: Registering a Histogram Metric

const Prometheus = require('prom-client');
const registry = new Prometheus.Registry();

const histogram = new Prometheus.Histogram({
  name: 'request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'status'],
  buckets: [0.1, 0.5, 1, 2, 5]
});

registry.registerMetric(histogram);

// Observe a value
histogram.observe({ method: 'POST', status: '201' }, 0.3);

8. Implementing Metric Labels

Label Name Description Usage Example Notes
job Identifies the job that is being monitored job="prometheus" Required for all metrics
instance Identifies the specific instance being monitored instance="localhost:9090" Useful for multi-instance setups
status Represents the status of a metric (e.g., success, error) status="success" Helps in filtering and alerting
method HTTP method used in a request (GET, POST, etc.) method="POST" Useful for API monitoring
endpoint API endpoint being monitored endpoint="/api/users" Essential for endpoint-specific metrics

Example: Adding Labels to a Counter Metric

const httpRequestCounter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'endpoint', 'status']
});

// Incrementing the counter with labels
httpRequestCounter.labels({method: 'GET', endpoint: '/api/users', status: 'success'}).inc();
Note: Label values should be kept to a minimum to avoid high cardinality issues.
Warning: Avoid using labels with high cardinality (e.g., user IDs, session tokens) as they can lead to performance issues.

9. Following Naming Conventions

When implementing custom metrics in Prometheus, following naming conventions ensures consistency, readability, and compatibility with the ecosystem.

Rule Description Example
Metric Name Must be lowercase and use underscores. Avoid spaces or hyphens. http_request_duration_seconds
Unit Suffixes Use standard unit suffixes for clarity (e.g., _seconds, _bytes). disk_usage_bytes
Label Names Use lowercase, underscores. Avoid special characters. status_code
Help Text Descriptive and concise. Use full words, not abbreviations. HTTP request duration in seconds
Note: Prometheus metric names must match the regex [a-zA-Z_:][a-zA-Z0-9_:]*. Avoid starting with a number or using colons in names unless necessary for namespace separation.

10. Using Pushgateway for Batch Jobs

The Pushgateway is a component that allows batch jobs to expose their metrics to Prometheus. It is particularly useful when the lifecycle of a job is short and it's not feasible to have the job directly scrape itself or push metrics via an HTTP endpoint.

Feature Description Notes
Pushgateway URL http://pushgateway:9091 Default address for the Pushgateway server
Job Name batch_job Unique identifier for the job pushing metrics
Instance Name batch_instance_1 Unique identifier for a specific instance of the job

Example: Pushing Metrics to Pushgateway

curl -X POST http://pushgateway:9091/metrics/job/batch_job/instance/batch_instance_1 \
  -d 'my_metric{label="value"} 42'
Note: The Pushgateway is not designed for high-frequency metric updates. It's best suited for batch jobs that run periodically.
Warning: Metrics pushed to the Pushgateway are not immediately scraped by Prometheus. They are stored until the next scrape or until the job is explicitly removed.

11. Implementing Metric Instrumentation

Instrumenting custom metrics in Prometheus involves defining and registering metrics within your application code. These metrics are exposed via an HTTP endpoint (typically /metrics) that Prometheus scrapes.

Example: Registering a Counter Metric

const client = require('prom-client');
const counter = new client.Counter({
  name: 'http_requests_total',
  help: 'Total number of HTTP requests',
  labelNames: ['method', 'code']
});

// Increment the counter
counter.inc({ method: 'GET', code: '200' });

Example: Registering a Histogram Metric

const histogram = new client.Histogram({
  name: 'request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  buckets: [0.1, 0.5, 1, 2, 5]
});

// Observe a value
histogram.observe(0.3);
Metric Type Description Usage Example Notes
Counter Monotonically increasing value, typically used for counts of events. counter.inc() Must only increase, resets on restart.
Gauge Can increase or decrease, represents a single numerical value. gauge.set(10) Useful for current state or measurements.
Histogram Tracks distribution of values, useful for latency or size measurements. histogram.observe(0.3) Supports configurable buckets.
Summary Similar to Histogram but with configurable quantiles. summary.observe(0.3) Quantiles are calculated client-side.
Note: All metrics must be registered before they can be used. Prometheus client libraries provide methods to register and expose these metrics via an HTTP endpoint.

12. Testing Custom Metrics

Example: Unit Test for Custom Metric

package main

import (
    "testing"
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/testutil"
)

var (
    customCounter = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "custom_requests_total",
            Help: "Total number of custom requests",
        },
        []string{"method", "endpoint"},
    )
)

func init() {
    prometheus.MustRegister(customCounter)
}

func TestCustomMetrics(t *testing.T) {
    customCounter.WithLabelValues("GET", "/api/users").Inc()
    customCounter.WithLabelValues("POST", "/api/users").Inc()

    expected := `# HELP custom_requests_total Total number of custom requests
# TYPE custom_requests_total counter
custom_requests_total{endpoint="/api/users",method="GET"} 1
custom_requests_total{endpoint="/api/users",method="POST"} 1
`
    if err := testutil.CollectAndCompare(customCounter, strings.NewReader(expected)); err != nil {
        t.Errorf("Unexpected metrics: %v", err)
    }
}
Metric Type Testing Method Notes
Counter testutil.CollectAndCompare() Use for verifying counter increments
Gauge testutil.CollectAndCompare() Use for verifying gauge values
Histogram testutil.CollectAndCompare() Use for verifying histogram buckets
Summary testutil.CollectAndCompare() Use for verifying summary quantiles
Note: The testutil package provides utilities for testing Prometheus metrics in Go.
Warning: Ensure that your test environment does not interfere with production metrics.

Testing Best Practices

  • Use testutil.CollectAndCompare for metric validation
  • Test with various label combinations to ensure coverage
  • Verify that metrics are properly registered before testing
  • Use strings.NewReader to provide expected metric output

Was this article helpful?