Implementing Custom Metrics
1. Using Client Libraries
Prometheus provides official client libraries for various programming languages to expose metrics from applications. These libraries handle the exposition of metrics in the Prometheus text format and provide convenient APIs for instrumenting code.
| Language | Library | Repository | Status |
|---|---|---|---|
| Go | github.com/prometheus/client_golang |
GitHub | NEW v1.19.0 |
| Java | io.prometheus.client |
GitHub | NEW v0.16.0 |
| Python | prometheus-client |
GitHub | NEW v0.19.0 |
| JavaScript | prom-client |
GitHub | NEW v15.0.0 |
| Ruby | prometheus-client |
GitHub | DEPRECATED v0.12.0 |
Example: Go Client Library
package main
import (
"log"
"net/http"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var (
httpRequestCount = promauto.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
},
[]string{"method", "code"},
)
)
func main() {
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
httpRequestCount.WithLabelValues(r.Method, "200").Inc()
w.Write([]byte("Hello, World!"))
})
log.Fatal(http.ListenAndServe(":8080", nil))
}
Example: Python Client Library
from prometheus_client import Counter, start_http_server
# Create a counter metric
http_requests_total = Counter(
"http_requests_total",
"Total number of HTTP requests",
[ "method", "code" ]
)
def handle_request():
# Increment the counter
http_requests_total.labels(method="GET", code="200").inc()
return "Hello, World!"
if __name__ == "__main__":
start_http_server(8080)
/metrics endpoint by default, which Prometheus scrapes.
2. Creating Counter Metrics
| Metric Name | Type | Description | Usage Example |
|---|---|---|---|
| http_requests_total | Counter | Total number of HTTP requests handled by the application | http_requests_total{method="GET",endpoint="/api/users"} 120 |
| error_count | Counter | Number of errors encountered in the system | error_count{type="database",code="500"} 5 |
Example: Registering a Counter in Go
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
httpRequestCount = promauto.NewCounterVec(prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total number of HTTP requests",
}, []string{"method", "endpoint"})
)
func main() {
// Increment counter on each request
httpRequestCount.WithLabelValues("GET", "/api/users").Inc()
}
3. Creating Gauge Metrics
| Property | Description | Example |
|---|---|---|
| gauge.name | Name of the gauge metric | http_requests_total |
| gauge.help | Help text describing the metric | Total number of HTTP requests |
| gauge.value | Current value of the gauge | 120 |
Example: Creating a Gauge Metric
const client = require('prom-client');
const gauge = new client.Gauge({
name: 'nodejs_heap_used_bytes',
help: 'The number of bytes of the Node.js heap used',
});
// Set gauge value
gauge.set(process.memoryUsage().heapUsed);
// Increment gauge value
gauge.inc(1024); // Adds 1KB to the gauge
// Decrement gauge value
gauge.dec(512); // Removes 512 bytes from the gauge
4. Creating Histogram Metrics
| Metric Name | Labels | Description | Usage |
|---|---|---|---|
| request_duration_seconds | method, endpoint |
Duration of HTTP requests in seconds | Used to monitor latency of API endpoints |
| response_size_bytes | status_code |
Size of HTTP responses in bytes | Tracks payload sizes for performance analysis |
Example: Histogram Metric Definition
const promClient = require('prom-client');
const histogram = new promClient.Histogram({
name: 'request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'endpoint'],
buckets: [0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10]
});
module.exports = histogram;
Example: Recording Histogram Values
const start = Date.now();
// Simulate request processing
await processRequest();
const duration = (Date.now() - start) / 1000;
histogram.observe({ method: 'GET', endpoint: '/api/users' }, duration);
5. Creating Summary Metrics
Summary metrics in Prometheus are used to track the distribution of events, such as request durations or response sizes. They allow you to calculate quantiles over a sliding time window.
| Property | Description | Default Value | Notes |
|---|---|---|---|
| name | The name of the summary metric | None | Metric names must follow Prometheus naming conventions |
| help | A help text describing the metric | None | Help text is used in Prometheus UI and documentation |
| label_names | List of label names to be included in the metric | None | Labels allow for grouping and filtering of metrics |
| max_age | Maximum age of observations to keep in memory | 60 seconds | Controls how long observations are retained before being discarded |
| buf_cap | Buffer capacity for observations | 1000 | Defines how many observations can be stored in the buffer |
Example: Creating a Summary Metric in Go
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
)
var (
requestDuration = promauto.NewSummary(prometheus.SummaryOpts{
Name: "http_request_duration_seconds",
Help: "Duration of HTTP requests in seconds",
LabelNames: []string{"method", "endpoint"},
MaxAge: 60 * time.Second,
BufCap: 1000,
})
)
func main() {
// Example usage
requestDuration.WithLabelValues("GET", "/api/users").Observe(0.5)
}
6. Exposing Metrics Endpoint
| Property | Description | Default Value | Notes |
|---|---|---|---|
| endpoint | URL path to expose metrics | /metrics |
Can be customized to any valid path |
| port | Port to bind the metrics server | 9090 |
Must be an available port on the host |
| handler | Function to handle metric requests | defaultHandler |
Can be replaced with custom logic |
Example: Basic Metrics Endpoint Setup
const client = require('prom-client');
const express = require('express');
const app = express();
const collectDefaultMetrics = client.collectDefaultMetrics;
const register = client.register;
// Collect default metrics
collectDefaultMetrics({ prefix: 'myapp_' });
app.get('/metrics', async (req, res) => {
res.set('Content-Type', register.contentType);
res.end(await register.metrics());
});
app.listen(9090, () => {
console.log('Metrics endpoint available at http://localhost:9090/metrics');
});
7. Using Metric Registry
| Property | Description | Default Value | Notes |
|---|---|---|---|
| registry | Instance of the Prometheus registry to register metrics with | null |
Must be an instance of Prometheus.Registry |
| name | Name of the metric to register | undefined |
Must be a valid Prometheus metric name |
| help | Help text for the metric | undefined |
Descriptive text for the metric |
| labels | Array of label names to include in the metric | [] |
Labels must be strings |
Example: Registering a Counter Metric
const Prometheus = require('prom-client');
const registry = new Prometheus.Registry();
const counter = new Prometheus.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'status']
});
registry.registerMetric(counter);
// Increment the counter
counter.inc({ method: 'GET', status: '200' });
Example: Registering a Histogram Metric
const Prometheus = require('prom-client');
const registry = new Prometheus.Registry();
const histogram = new Prometheus.Histogram({
name: 'request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
labelNames: ['method', 'status'],
buckets: [0.1, 0.5, 1, 2, 5]
});
registry.registerMetric(histogram);
// Observe a value
histogram.observe({ method: 'POST', status: '201' }, 0.3);
8. Implementing Metric Labels
| Label Name | Description | Usage Example | Notes |
|---|---|---|---|
| job | Identifies the job that is being monitored | job="prometheus" |
Required for all metrics |
| instance | Identifies the specific instance being monitored | instance="localhost:9090" |
Useful for multi-instance setups |
| status | Represents the status of a metric (e.g., success, error) | status="success" |
Helps in filtering and alerting |
| method | HTTP method used in a request (GET, POST, etc.) | method="POST" |
Useful for API monitoring |
| endpoint | API endpoint being monitored | endpoint="/api/users" |
Essential for endpoint-specific metrics |
Example: Adding Labels to a Counter Metric
const httpRequestCounter = new client.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'endpoint', 'status']
});
// Incrementing the counter with labels
httpRequestCounter.labels({method: 'GET', endpoint: '/api/users', status: 'success'}).inc();
9. Following Naming Conventions
When implementing custom metrics in Prometheus, following naming conventions ensures consistency, readability, and compatibility with the ecosystem.
| Rule | Description | Example |
|---|---|---|
| Metric Name | Must be lowercase and use underscores. Avoid spaces or hyphens. | http_request_duration_seconds |
| Unit Suffixes | Use standard unit suffixes for clarity (e.g., _seconds, _bytes). |
disk_usage_bytes |
| Label Names | Use lowercase, underscores. Avoid special characters. | status_code |
| Help Text | Descriptive and concise. Use full words, not abbreviations. | HTTP request duration in seconds |
[a-zA-Z_:][a-zA-Z0-9_:]*. Avoid starting with a number or using colons in names unless necessary for namespace separation.
10. Using Pushgateway for Batch Jobs
The Pushgateway is a component that allows batch jobs to expose their metrics to Prometheus. It is particularly useful when the lifecycle of a job is short and it's not feasible to have the job directly scrape itself or push metrics via an HTTP endpoint.
| Feature | Description | Notes |
|---|---|---|
| Pushgateway URL | http://pushgateway:9091 |
Default address for the Pushgateway server |
| Job Name | batch_job |
Unique identifier for the job pushing metrics |
| Instance Name | batch_instance_1 |
Unique identifier for a specific instance of the job |
Example: Pushing Metrics to Pushgateway
curl -X POST http://pushgateway:9091/metrics/job/batch_job/instance/batch_instance_1 \
-d 'my_metric{label="value"} 42'
11. Implementing Metric Instrumentation
Instrumenting custom metrics in Prometheus involves defining and registering metrics within your application code. These metrics are exposed via an HTTP endpoint (typically /metrics) that Prometheus scrapes.
Example: Registering a Counter Metric
const client = require('prom-client');
const counter = new client.Counter({
name: 'http_requests_total',
help: 'Total number of HTTP requests',
labelNames: ['method', 'code']
});
// Increment the counter
counter.inc({ method: 'GET', code: '200' });
Example: Registering a Histogram Metric
const histogram = new client.Histogram({
name: 'request_duration_seconds',
help: 'Duration of HTTP requests in seconds',
buckets: [0.1, 0.5, 1, 2, 5]
});
// Observe a value
histogram.observe(0.3);
| Metric Type | Description | Usage Example | Notes |
|---|---|---|---|
| Counter | Monotonically increasing value, typically used for counts of events. | counter.inc() |
Must only increase, resets on restart. |
| Gauge | Can increase or decrease, represents a single numerical value. | gauge.set(10) |
Useful for current state or measurements. |
| Histogram | Tracks distribution of values, useful for latency or size measurements. | histogram.observe(0.3) |
Supports configurable buckets. |
| Summary | Similar to Histogram but with configurable quantiles. | summary.observe(0.3) |
Quantiles are calculated client-side. |
12. Testing Custom Metrics
Example: Unit Test for Custom Metric
package main
import (
"testing"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/testutil"
)
var (
customCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "custom_requests_total",
Help: "Total number of custom requests",
},
[]string{"method", "endpoint"},
)
)
func init() {
prometheus.MustRegister(customCounter)
}
func TestCustomMetrics(t *testing.T) {
customCounter.WithLabelValues("GET", "/api/users").Inc()
customCounter.WithLabelValues("POST", "/api/users").Inc()
expected := `# HELP custom_requests_total Total number of custom requests
# TYPE custom_requests_total counter
custom_requests_total{endpoint="/api/users",method="GET"} 1
custom_requests_total{endpoint="/api/users",method="POST"} 1
`
if err := testutil.CollectAndCompare(customCounter, strings.NewReader(expected)); err != nil {
t.Errorf("Unexpected metrics: %v", err)
}
}
| Metric Type | Testing Method | Notes |
|---|---|---|
| Counter | testutil.CollectAndCompare() |
Use for verifying counter increments |
| Gauge | testutil.CollectAndCompare() |
Use for verifying gauge values |
| Histogram | testutil.CollectAndCompare() |
Use for verifying histogram buckets |
| Summary | testutil.CollectAndCompare() |
Use for verifying summary quantiles |
testutil package provides utilities for testing Prometheus metrics in Go.