Using Pushgateway
1. Installing Pushgateway
Example: Download and Run Pushgateway
curl -LO https://github.com/prometheus/pushgateway/releases/download/v1.8.0/pushgateway-1.8.0.linux-amd64.tar.gz
tar xvfz pushgateway-1.8.0.linux-amd64.tar.gz
cd pushgateway-1.8.0.linux-amd64
./pushgateway
| Installation Method | Command | Notes |
|---|---|---|
| Binary Download | curl -LO https://github.com/prometheus/pushgateway/releases/download/v1.8.0/pushgateway-1.8.0.linux-amd64.tar.gz |
Download latest stable release for Linux AMD64 |
| Docker | docker run -d --name pushgateway prom/pushgateway |
Run Pushgateway in a Docker container |
| Homebrew (macOS) | brew install pushgateway |
Install via Homebrew on macOS |
| Systemd Service | systemctl enable pushgateway.service |
Enable Pushgateway as a systemd service |
Note: Pushgateway does not persist metrics across restarts by default. Use
--storage.tsdb.retention.time to configure persistence.
2. Pushing Metrics to Gateway
| Field | Description | Example |
|---|---|---|
| job | Job name to associate with the metrics | my-job |
| instance | Instance identifier (optional) | instance-1 |
| metrics | Metrics to push (in Prometheus text format) | |
Example: Pushing Metrics to Pushgateway
curl -X PUT \
--data-binary @metrics.txt \
http://pushgateway:9091/metrics/job/my-job/instance/instance-1
Note: Pushgateway is not designed for high-frequency metric pushing. It's best suited for batch jobs or services that run for a limited time.
3. Using Client Libraries
Prometheus client libraries provide a convenient way to instrument your applications and push metrics to a Pushgateway. These libraries support multiple programming languages and offer abstractions for common metric types.
| Language | Library Name | Pushgateway Support | Notes |
|---|---|---|---|
| Go | github.com/prometheus/client_golang |
Yes | Official Prometheus Go client with Pushgateway support via prometheus/push package. |
| Java | io.prometheus.client |
Yes | Official client supports Pushgateway via PushGateway class. |
| Python | prometheus-client |
Yes | Official Python client includes PushGateway support. |
| Node.js | prom-client |
Yes | Supports Pushgateway via Pushgateway class. |
| Ruby | prometheus-client |
Yes | Official Ruby client supports Pushgateway. |
| C++ | prometheus-cpp |
Yes | Supports Pushgateway via Prometheus::PushGateway. |
Example: Pushing Metrics from Go
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/push"
)
func main() {
counter := prometheus.NewCounter(prometheus.CounterOpts{
Name: "requests_total",
Help: "Total number of requests.",
})
prometheus.MustRegister(counter)
counter.Inc()
err := push.AddFromGatherer(
"my_job",
"http://localhost:9091",
prometheus.Gatherers{prometheus.DefaultGatherer},
)
if err != nil {
panic(err)
}
}
Example: Pushing Metrics from Python
from prometheus_client import CollectorRegistry, Gauge, push_to_gateway
registry = CollectorRegistry()
gauge = Gauge("requests_total", "Total number of requests.", registry=registry)
gauge.set(42)
push_to_gateway("localhost:9091", "my_job", registry)
Note: Client libraries typically handle the serialization and transport of metrics to the Pushgateway. Ensure that your application has appropriate error handling for network or gateway issues.
4. Configuring Job and Instance Labels
| Label Type | Configuration Method | Description | Example |
|---|---|---|---|
| job | pushgateway --job |
Defines the job name for metrics | pushgateway --job=service-metrics |
| instance | pushgateway --instance |
Sets the instance identifier for metrics | pushgateway --instance=web01 |
| job_label | Pushgateway API | Used in API calls to set job label | curl -X PUT "http://pushgateway:9091/metrics/job/myjob/instance/myinstance" |
| instance_label | Pushgateway API | Used in API calls to set instance label | curl -X PUT "http://pushgateway:9091/metrics/job/myjob/instance/myinstance" |
Example: Setting Labels via HTTP PUT Request
curl -X PUT "http://pushgateway:9091/metrics/job/api-server/instance/web01" --data-binary @metrics.txt
5. Deleting Metrics from Gateway
| Operation | Description | Notes |
|---|---|---|
| DELETE /metrics/name/{name} | Deletes all metrics with the given name from the Pushgateway. | Requires appropriate authentication if enabled. |
| DELETE /metrics/job/{job}/name/{name} | Deletes metrics with the given name and job from the Pushgateway. | Useful for cleaning up specific metric sets. |
| DELETE /metrics/job/{job}/instance/{instance}/name/{name} | Deletes a specific metric by name, job, and instance. | Most granular deletion option. |
Example: Delete a metric by name
curl -X DELETE http://pushgateway.example.com:9091/metrics/name/my_metric
Example: Delete a metric by job and name
curl -X DELETE http://pushgateway.example.com:9091/metrics/job/my_job/name/my_metric
Note: The Pushgateway does not support deletion of individual metric families or partial deletion of metrics. You must delete by name, job, or full instance path.
6. Configuring Persistence
| Option | Values | Notes |
|---|---|---|
| storage.tsdb.path | data/ |
Directory to store the TSDB (Time Series Database) data |
| storage.tsdb.retention.time | 15d |
How long to retain samples in the database |
| storage.tsdb.retention.size | 512MB |
Maximum size of the database before old data is deleted |
| storage.tsdb.wal.dir | data/wal/ |
Directory to store the Write-Ahead Log (WAL) |
Example: Prometheus Configuration with Persistence Settings
storage:
tsdb:
path: "data/"
retention:
time: 15d
size: 512MB
wal:
dir: "data/wal/"
Note: The
storage.tsdb.retention.time and storage.tsdb.retention.size settings are mutually exclusive. If both are set, time takes precedence.
Warning: Ensure sufficient disk space is available for the configured retention settings to prevent data loss.
7. Scraping Pushgateway from Prometheus
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| scrape_configs | Defines how to scrape targets including Pushgateway | None | Must be defined in Prometheus config file |
| job_name | Unique identifier for the scraping job | None | Required field in scrape config |
| static_configs | Defines static targets to scrape | None | Used to define Pushgateway endpoint |
| targets | List of Pushgateway endpoints to scrape | None | Must include full URL with port |
| params | Query parameters to pass during scraping | None | Optional, used for filtering metrics |
Example: Basic Pushgateway Scrape Configuration
scrape_configs:
- job_name: 'pushgateway'
static_configs:
- targets: ['localhost:9091']
Example: Pushgateway Scrape with Filtering
scrape_configs:
- job_name: 'pushgateway'
static_configs:
- targets: ['localhost:9091']
params:
'match[]': ['{job="myapp"}']
Note: Pushgateway must be running and accessible from the Prometheus server for scraping to work.
Warning: Pushgateway is not designed for high-volume scraping; it's intended for occasional metric pushes.
8. Understanding Push vs Pull Trade-offs
| Aspect | Pushgateway | Pull (Direct) |
|---|---|---|
| Data Collection | Services push metrics to Pushgateway | Pushgateway pulls metrics from services |
| Latency | Higher due to push overhead | Lower, real-time scraping |
| Reliability | Requires handling of transient failures | More resilient to temporary outages |
| Use Case | Batch jobs, ephemeral services | Long-running services, stable systems |
| Configuration | Requires Pushgateway instance | No additional gateway required |
| Scalability | Can be a bottleneck under high load | Better scalability for large deployments |
Pushgateway vs Direct Pull
- Pushgateway: Ideal for batch jobs or services that don't run continuously. Metrics are pushed to a central Pushgateway, which is then scraped by Prometheus.
- Direct Pull: Prometheus actively scrapes metrics from targets. Preferred for long-running services and systems where direct access is feasible.
Note: Pushgateway is not a replacement for Prometheus scraping. It's an alternative when direct scraping isn't possible or practical.
9. Implementing Batch Job Metrics
When using the Pushgateway for batch jobs, you must ensure that metrics are properly pushed and cleaned up after job completion.
| Component | Description | Notes |
|---|---|---|
| pushgateway.url | http://localhost:9091 |
URL of the Pushgateway instance |
| job.name | batch-job-1 |
Unique identifier for the job |
| metrics.endpoint | /metrics |
Endpoint to push metrics to |
Example: Pushing Metrics from a Batch Job
curl -X POST http://localhost:9091/metrics/job/batch-job-1 \
-d 'metric_name{label="value"} 42'
Note: Metrics pushed to the Pushgateway are retained until explicitly deleted or replaced by a new push from the same job.
Warning: If a batch job fails or is interrupted, ensure that the Pushgateway metrics are cleaned up to avoid stale data.
10. Using Pushgateway with Cron Jobs
Pushgateway is useful for collecting metrics from batch jobs or cron tasks that run periodically and do not maintain long-lived connections to Prometheus.
| Configuration | Description | Example |
|---|---|---|
| pushgateway.url | URL of the Pushgateway instance | http://localhost:9091 |
| job.name | Unique identifier for the job | backup-script |
| push.interval | How often to push metrics (e.g., every 5 minutes) | 5m |
Example: Pushing Metrics from a Cron Job
#!/bin/bash
# Collect metrics and push to Pushgateway
METRICS_FILE="/tmp/metrics.txt"
echo "backup_success{job="backup-script"} 1" > $METRICS_FILE
echo "backup_duration_seconds{job="backup-script"} 30.5" >> $METRICS_FILE
# Push to Pushgateway
curl -X POST http://localhost:9091/metrics/job/backup-script < $METRICS_FILE
Note: Metrics pushed with the same job and instance labels will overwrite previous ones. Ensure unique instance labels if multiple jobs run concurrently.
11. Avoiding Pushgateway Pitfalls
Note: The Pushgateway is not suitable for all use cases. It's designed for batch jobs or short-lived services, not for continuous metric collection.
| Pitfall | Description | Best Practice |
|---|---|---|
| Frequent Pushes | Pushing metrics too often can overwhelm the Pushgateway and cause performance issues. | Batch metric updates and push only when necessary. Use push.add for periodic updates. |
| Unlabeled Metrics | Metrics pushed without proper labels can lead to data inconsistency or loss. | Always label metrics with meaningful identifiers like job name, instance, and environment. |
| Pushgateway as a Long-Term Store | Using Pushgateway for long-term metric storage is not recommended due to its in-memory nature. | Use Prometheus or another time-series database for persistent metric storage. Pushgateway is meant for temporary aggregation. |
| Race Conditions | Concurrent pushes from multiple processes or threads can lead to data loss or corruption. | Implement proper locking or use atomic operations when pushing metrics from concurrent sources. |
Example: Proper Labeling with Pushgateway
const client = require('prom-client');
const Pushgateway = require('prom-client/pushgateway');
// Create a counter metric
const counter = new client.Counter({
name: 'requests_total',
help: 'Total number of requests',
labelNames: ['method', 'status']
});
// Add labels before pushing
counter.labels('GET', '200').inc();
// Push to Pushgateway with job name and instance
const pushgateway = new Pushgateway('http://localhost:9091');
pushgateway.pushAdd({ jobName: 'batch-job', groupings: { instance: 'instance-1' } }, client.register.metrics())
.then(() => console.log('Metrics pushed successfully'))
.catch(err => console.error('Failed to push metrics:', err));
Warning: Never use Pushgateway for real-time metrics or services that run continuously. It's intended for batch jobs or ephemeral services.
12. Monitoring Pushgateway Health
| Health Check Endpoint | URL | Description |
|---|---|---|
| /-/healthy | http://pushgateway:9091/-/healthy |
Indicates if the Pushgateway is healthy and ready to accept pushes. |
| /-/ready | http://pushgateway:9091/-/ready |
Indicates if the Pushgateway is ready to serve requests. |
Note: These endpoints return a 200 OK status code if the service is healthy or ready, and a 503 Service Unavailable status code if not.