Skip to main content

Using Pushgateway

1. Installing Pushgateway

Example: Download and Run Pushgateway

curl -LO https://github.com/prometheus/pushgateway/releases/download/v1.8.0/pushgateway-1.8.0.linux-amd64.tar.gz
tar xvfz pushgateway-1.8.0.linux-amd64.tar.gz
cd pushgateway-1.8.0.linux-amd64
./pushgateway
Installation Method Command Notes
Binary Download curl -LO https://github.com/prometheus/pushgateway/releases/download/v1.8.0/pushgateway-1.8.0.linux-amd64.tar.gz Download latest stable release for Linux AMD64
Docker docker run -d --name pushgateway prom/pushgateway Run Pushgateway in a Docker container
Homebrew (macOS) brew install pushgateway Install via Homebrew on macOS
Systemd Service systemctl enable pushgateway.service Enable Pushgateway as a systemd service
Note: Pushgateway does not persist metrics across restarts by default. Use --storage.tsdb.retention.time to configure persistence.

2. Pushing Metrics to Gateway

Field Description Example
job Job name to associate with the metrics my-job
instance Instance identifier (optional) instance-1
metrics Metrics to push (in Prometheus text format)
# TYPE my_metric counter
my_metric{label="value"} 123

Example: Pushing Metrics to Pushgateway

curl -X PUT \
  --data-binary @metrics.txt \
  http://pushgateway:9091/metrics/job/my-job/instance/instance-1
Note: Pushgateway is not designed for high-frequency metric pushing. It's best suited for batch jobs or services that run for a limited time.

3. Using Client Libraries

Prometheus client libraries provide a convenient way to instrument your applications and push metrics to a Pushgateway. These libraries support multiple programming languages and offer abstractions for common metric types.

Language Library Name Pushgateway Support Notes
Go github.com/prometheus/client_golang Yes Official Prometheus Go client with Pushgateway support via prometheus/push package.
Java io.prometheus.client Yes Official client supports Pushgateway via PushGateway class.
Python prometheus-client Yes Official Python client includes PushGateway support.
Node.js prom-client Yes Supports Pushgateway via Pushgateway class.
Ruby prometheus-client Yes Official Ruby client supports Pushgateway.
C++ prometheus-cpp Yes Supports Pushgateway via Prometheus::PushGateway.

Example: Pushing Metrics from Go

package main

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/push"
)

func main() {
    counter := prometheus.NewCounter(prometheus.CounterOpts{
        Name: "requests_total",
        Help: "Total number of requests.",
    })

    prometheus.MustRegister(counter)
    counter.Inc()

    err := push.AddFromGatherer(
        "my_job",
        "http://localhost:9091",
        prometheus.Gatherers{prometheus.DefaultGatherer},
    )
    if err != nil {
        panic(err)
    }
}

Example: Pushing Metrics from Python

from prometheus_client import CollectorRegistry, Gauge, push_to_gateway

registry = CollectorRegistry()
gauge = Gauge("requests_total", "Total number of requests.", registry=registry)
gauge.set(42)

push_to_gateway("localhost:9091", "my_job", registry)
Note: Client libraries typically handle the serialization and transport of metrics to the Pushgateway. Ensure that your application has appropriate error handling for network or gateway issues.

4. Configuring Job and Instance Labels

Label Type Configuration Method Description Example
job pushgateway --job Defines the job name for metrics pushgateway --job=service-metrics
instance pushgateway --instance Sets the instance identifier for metrics pushgateway --instance=web01
job_label Pushgateway API Used in API calls to set job label curl -X PUT "http://pushgateway:9091/metrics/job/myjob/instance/myinstance"
instance_label Pushgateway API Used in API calls to set instance label curl -X PUT "http://pushgateway:9091/metrics/job/myjob/instance/myinstance"

Example: Setting Job and Instance Labels via CLI

pushgateway --job=web-server --instance=prod-01

Example: Setting Labels via HTTP PUT Request

curl -X PUT "http://pushgateway:9091/metrics/job/api-server/instance/web01" --data-binary @metrics.txt

5. Deleting Metrics from Gateway

Operation Description Notes
DELETE /metrics/name/{name} Deletes all metrics with the given name from the Pushgateway. Requires appropriate authentication if enabled.
DELETE /metrics/job/{job}/name/{name} Deletes metrics with the given name and job from the Pushgateway. Useful for cleaning up specific metric sets.
DELETE /metrics/job/{job}/instance/{instance}/name/{name} Deletes a specific metric by name, job, and instance. Most granular deletion option.

Example: Delete a metric by name

curl -X DELETE http://pushgateway.example.com:9091/metrics/name/my_metric

Example: Delete a metric by job and name

curl -X DELETE http://pushgateway.example.com:9091/metrics/job/my_job/name/my_metric
Note: The Pushgateway does not support deletion of individual metric families or partial deletion of metrics. You must delete by name, job, or full instance path.

6. Configuring Persistence

Option Values Notes
storage.tsdb.path data/ Directory to store the TSDB (Time Series Database) data
storage.tsdb.retention.time 15d How long to retain samples in the database
storage.tsdb.retention.size 512MB Maximum size of the database before old data is deleted
storage.tsdb.wal.dir data/wal/ Directory to store the Write-Ahead Log (WAL)

Example: Prometheus Configuration with Persistence Settings

storage:
  tsdb:
    path: "data/"
    retention:
      time: 15d
      size: 512MB
    wal:
      dir: "data/wal/"
Note: The storage.tsdb.retention.time and storage.tsdb.retention.size settings are mutually exclusive. If both are set, time takes precedence.
Warning: Ensure sufficient disk space is available for the configured retention settings to prevent data loss.

7. Scraping Pushgateway from Prometheus

Configuration Option Description Default Value Notes
scrape_configs Defines how to scrape targets including Pushgateway None Must be defined in Prometheus config file
job_name Unique identifier for the scraping job None Required field in scrape config
static_configs Defines static targets to scrape None Used to define Pushgateway endpoint
targets List of Pushgateway endpoints to scrape None Must include full URL with port
params Query parameters to pass during scraping None Optional, used for filtering metrics

Example: Basic Pushgateway Scrape Configuration

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['localhost:9091']

Example: Pushgateway Scrape with Filtering

scrape_configs:
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['localhost:9091']
    params:
      'match[]': ['{job="myapp"}']
Note: Pushgateway must be running and accessible from the Prometheus server for scraping to work.
Warning: Pushgateway is not designed for high-volume scraping; it's intended for occasional metric pushes.

8. Understanding Push vs Pull Trade-offs

Aspect Pushgateway Pull (Direct)
Data Collection Services push metrics to Pushgateway Pushgateway pulls metrics from services
Latency Higher due to push overhead Lower, real-time scraping
Reliability Requires handling of transient failures More resilient to temporary outages
Use Case Batch jobs, ephemeral services Long-running services, stable systems
Configuration Requires Pushgateway instance No additional gateway required
Scalability Can be a bottleneck under high load Better scalability for large deployments

Pushgateway vs Direct Pull

  • Pushgateway: Ideal for batch jobs or services that don't run continuously. Metrics are pushed to a central Pushgateway, which is then scraped by Prometheus.
  • Direct Pull: Prometheus actively scrapes metrics from targets. Preferred for long-running services and systems where direct access is feasible.
Note: Pushgateway is not a replacement for Prometheus scraping. It's an alternative when direct scraping isn't possible or practical.

9. Implementing Batch Job Metrics

When using the Pushgateway for batch jobs, you must ensure that metrics are properly pushed and cleaned up after job completion.

Component Description Notes
pushgateway.url http://localhost:9091 URL of the Pushgateway instance
job.name batch-job-1 Unique identifier for the job
metrics.endpoint /metrics Endpoint to push metrics to

Example: Pushing Metrics from a Batch Job

curl -X POST http://localhost:9091/metrics/job/batch-job-1 \
  -d 'metric_name{label="value"} 42'
Note: Metrics pushed to the Pushgateway are retained until explicitly deleted or replaced by a new push from the same job.
Warning: If a batch job fails or is interrupted, ensure that the Pushgateway metrics are cleaned up to avoid stale data.

10. Using Pushgateway with Cron Jobs

Pushgateway is useful for collecting metrics from batch jobs or cron tasks that run periodically and do not maintain long-lived connections to Prometheus.

Configuration Description Example
pushgateway.url URL of the Pushgateway instance http://localhost:9091
job.name Unique identifier for the job backup-script
push.interval How often to push metrics (e.g., every 5 minutes) 5m

Example: Pushing Metrics from a Cron Job

#!/bin/bash
# Collect metrics and push to Pushgateway
METRICS_FILE="/tmp/metrics.txt"
echo "backup_success{job="backup-script"} 1" > $METRICS_FILE
echo "backup_duration_seconds{job="backup-script"} 30.5" >> $METRICS_FILE

# Push to Pushgateway
curl -X POST http://localhost:9091/metrics/job/backup-script < $METRICS_FILE
Note: Metrics pushed with the same job and instance labels will overwrite previous ones. Ensure unique instance labels if multiple jobs run concurrently.

11. Avoiding Pushgateway Pitfalls

Note: The Pushgateway is not suitable for all use cases. It's designed for batch jobs or short-lived services, not for continuous metric collection.
Pitfall Description Best Practice
Frequent Pushes Pushing metrics too often can overwhelm the Pushgateway and cause performance issues. Batch metric updates and push only when necessary. Use push.add for periodic updates.
Unlabeled Metrics Metrics pushed without proper labels can lead to data inconsistency or loss. Always label metrics with meaningful identifiers like job name, instance, and environment.
Pushgateway as a Long-Term Store Using Pushgateway for long-term metric storage is not recommended due to its in-memory nature. Use Prometheus or another time-series database for persistent metric storage. Pushgateway is meant for temporary aggregation.
Race Conditions Concurrent pushes from multiple processes or threads can lead to data loss or corruption. Implement proper locking or use atomic operations when pushing metrics from concurrent sources.

Example: Proper Labeling with Pushgateway

const client = require('prom-client');
const Pushgateway = require('prom-client/pushgateway');

// Create a counter metric
const counter = new client.Counter({
  name: 'requests_total',
  help: 'Total number of requests',
  labelNames: ['method', 'status']
});

// Add labels before pushing
counter.labels('GET', '200').inc();

// Push to Pushgateway with job name and instance
const pushgateway = new Pushgateway('http://localhost:9091');
pushgateway.pushAdd({ jobName: 'batch-job', groupings: { instance: 'instance-1' } }, client.register.metrics())
  .then(() => console.log('Metrics pushed successfully'))
  .catch(err => console.error('Failed to push metrics:', err));
Warning: Never use Pushgateway for real-time metrics or services that run continuously. It's intended for batch jobs or ephemeral services.

12. Monitoring Pushgateway Health

Health Check Endpoint URL Description
/-/healthy http://pushgateway:9091/-/healthy Indicates if the Pushgateway is healthy and ready to accept pushes.
/-/ready http://pushgateway:9091/-/ready Indicates if the Pushgateway is ready to serve requests.

Example: Checking Pushgateway Health

curl http://pushgateway:9091/-/healthy
Note: These endpoints return a 200 OK status code if the service is healthy or ready, and a 503 Service Unavailable status code if not.

Was this article helpful?