Setting Up Federation
1. Configuring Federate Endpoint
| Property | Description | Default Value |
|---|---|---|
| web.federate | Enables the federation endpoint | false |
| web.federate.path | Path to the federation endpoint | /federate |
| web.federate.limits | Maximum number of series to return from a single query | 100000 |
Note: The federate endpoint is used to scrape metrics from other Prometheus instances.
2. Selecting Metrics for Federation
Note: When federating metrics, it's crucial to select only the relevant metrics that are needed by downstream systems to avoid unnecessary load and data transfer.
| Metrics Type | Description | Federation Considerations |
|---|---|---|
| Counter | Tracks a monotonically increasing value, typically used for counting events or operations. | Include counters that are essential for monitoring system health and performance. Avoid including high-cardinality counters unless necessary. |
| Gauge | Represents a single numerical value that can be set or modified, useful for tracking metrics like memory usage or CPU load. | Only include gauges that are critical for operational visibility. Avoid including metrics with frequent updates or high cardinality. |
| Histogram | Measures the distribution of values in a set of samples, often used for latency or response time tracking. | Include histogram metrics that are relevant to service-level objectives (SLOs) and performance monitoring. Consider aggregating histogram buckets if needed. |
| Summary | Similar to histogram, but provides quantiles over a sliding time window. | Use summaries only if they are required for specific SLOs or SLIs. Avoid including summaries with high cardinality labels. |
Example: Selecting Metrics for Federation
# Example federation configuration to select specific metrics
scrape_configs:
- job_name: "federated-metrics"
honor_labels: true
metrics_path: /federate
params:
match[]:
- "{__name__=~"prometheus_http_requests_total|node_cpu_seconds_total"}"
- "{__name__=~"http_request_duration_seconds_bucket"}"
static_configs:
- targets: [ "localhost:9090" ]
Best Practices for Metric Selection
- Minimize Cardinality: Avoid including metrics with many unique label values.
- Focus on Relevance: Only include metrics that are essential for downstream monitoring systems.
- Use Regex Matching: Leverage Prometheus's regex matching to filter metrics efficiently.
- Monitor Federation Performance: Keep an eye on the federation endpoint's performance and adjust metric selection accordingly.
3. Implementing Hierarchical Federation
Hierarchical federation allows you to organize Prometheus instances in a tree-like structure, where child instances scrape metrics and forward them to parent instances. This setup is useful for scaling and organizing large monitoring environments.
| Component | Description | Configuration |
|---|---|---|
| prometheus.yml | Configuration file for each Prometheus instance | scrape_configs, federation |
| remote_write | Writes metrics to a remote endpoint (parent instance) | url, write_relabel_configs |
| remote_read | Reads metrics from a remote endpoint (child instance) | url, read_relabel_configs |
Example: Federation Configuration
# Child instance config
scrape_configs:
- job_name: "child"
static_configs:
- targets: ["localhost:9090"]
remote_write:
- url: "http://parent-instance:9091/api/v1/write"
write_relabel_configs:
- source_labels: [__name__]
regex: "up"
action: keep
# Parent instance config
scrape_configs:
- job_name: "federated"
federate:
paths:
- /federate
params:
match[]:
- "{job='child'}"
Note: Ensure that the
federate endpoint is properly configured on the child instance to allow metric federation.
Key Concepts
- Federate metrics from child instances to parent instances.
- Use
remote_writeto send metrics to a parent instance. - Use
remote_readto fetch metrics from a parent instance. - Configure
federatepaths to expose metrics for federation.
4. Using Cross-Service Federation
Prometheus federation allows you to aggregate metrics from multiple Prometheus instances into a single instance, typically used for cross-service or cross-cluster monitoring.
| Configuration | Description | Example |
|---|---|---|
| federation | Enables federation endpoint | true |
| remote_write | Configures remote write to send metrics to another Prometheus instance | {"url": "http://prometheus-remote:9090/api/v1/write"} |
Example: Federation Configuration
scrape_configs:
- job_name: "federate"
scrape_interval: 15s
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{job=~".*"}'
static_configs:
- targets: ["prometheus-main:9090"]
Note: The
honor_labels setting ensures that labels from the source Prometheus instance are preserved during federation.
Example: Remote Write Configuration
remote_write:
- url: "http://prometheus-remote:9090/api/v1/write"
queue_config:
capacity: 1000
max_samples_per_send: 100
Key Points
- Federation allows aggregation of metrics from multiple Prometheus instances.
- Use
honor_labelsto preserve label integrity during federation. - Remote write can be used to send federated metrics to another Prometheus instance for centralized monitoring.
5. Configuring Honor Labels
| Label | Description | Default Value | Notes |
|---|---|---|---|
| honor_labels | Controls whether Prometheus should honor the label values from the scraped data | false |
When set to true, Prometheus will not override labels from the scraped data with those from the scrape configuration |
Example: Configuring honor_labels in Prometheus scrape configuration
scrape_configs:
- job_name: "prometheus"
honor_labels: true
static_configs:
- targets: ["localhost:9090"]
Note: Setting
honor_labels to true can lead to label conflicts if the same label name exists in both the target and the scrape configuration. It's important to ensure that labels are consistent across configurations to avoid unexpected behavior.
6. Setting Scrape Federation Targets
| Field | Description | Example |
|---|---|---|
| job_name | Unique identifier for the scrape job | prometheus-federation |
| honor_labels | Whether to honor labels from scraped targets | true |
| scrape_interval | How often to scrape targets | 15s |
| scrape_timeout | Timeout for individual scrapes | 10s |
| metrics_path | Path to scrape metrics from | /federate |
| relabel_configs | Rules to modify labels before scraping | [{source_labels: [__address__], target_label: job}] |
Example: Federation Configuration
scrape_configs:
- job_name: 'prometheus-federation'
honor_labels: true
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: /federate
relabel_configs:
- source_labels: [__address__]
target_label: job
static_configs:
- targets:
- 'prometheus1:9090'
- 'prometheus2:9090'
Note: The
/federate endpoint must be enabled on the target Prometheus instances to allow federation.
Warning: Federation can increase network traffic significantly. Ensure that your network bandwidth can handle the load.
7. Implementing Global View
Example: Federation Configuration
# prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ['localhost:9090']
- job_name: "federate"
scrape_interval: 30s
honor_labels: true
metrics_path: /federate
params:
match[]:
- '{job="prometheus"}'
static_configs:
- targets: ['localhost:9091']
| Configuration | Description | Values | Notes |
|---|---|---|---|
| job_name | Name of the job | federate |
Used to identify the federation job |
| metrics_path | Endpoint to scrape metrics from | /federate |
Must be set to /federate for federation |
| honor_labels | Whether to honor labels from scraped metrics | true |
Prevents label conflicts in federated metrics |
| match[] | Metrics to include in federation | {job="prometheus"} |
Allows filtering of metrics to federate |
Note: The
/federate endpoint is specific to Prometheus and must be used for federation.Warning: Ensure that the
match[] parameter is correctly configured to avoid scraping too many or too few metrics.Global View Implementation Steps
- Configure Prometheus instances to scrape metrics from other instances
- Set up the
/federateendpoint on each instance - Define job names and match patterns to control which metrics are federated
- Ensure
honor_labelsis enabled to prevent label conflicts
8. Using External Labels
External labels are used to add additional metadata to metrics collected by Prometheus. They help in identifying the source of the metrics and can be useful for federation and multi-tenancy setups.
| Label | Description | Usage |
|---|---|---|
| external.label.name | Name of the external label to be added | Used in configuration files or command-line arguments |
| external.label.value | Value of the external label to be added | Can be static or dynamic based on environment variables |
Example: Configuring External Labels in Prometheus
global:
external_labels:
region: us-west-1
environment: production
team: backend
Note: External labels are applied to all metrics scraped by the Prometheus instance and are particularly useful when federating metrics across multiple instances or environments.
9. Optimizing Federation Queries
Example: Using PromQL to Reduce Federation Load
# Instead of federating all metrics, filter at the source
up{job="prometheus"} == 1
# Use rate() to reduce cardinality in federated queries
rate(prometheus_http_requests_total[5m])
# Apply label filters to limit scope
up{job="prometheus", instance=~"server.*"} == 1| Optimization | Description | Impact |
|---|---|---|
| Query Filtering | Filter metrics at source before federation | Reduces bandwidth and processing load |
| Rate Functions | Use rate() and irate() to aggregate |
Improves query performance in federated setups |
| Label Selectors | Limit scope with instance, job filters |
Reduces cardinality and improves response times |
Note: Federation queries should be optimized to avoid over-fetching data. Use
match[] patterns carefully to prevent excessive metric collection.
Example: Federation Rule with Query Optimization
# Prometheus federation config with optimized queries
rule_files:
- "federation_rules.yml"
scrape_configs:
- job_name: "federated-prometheus"
scrape_interval: 30s
metrics_path: "/federate"
params:
"match[]":
- "{__name__=~\"prometheus_.*\"}"
- "{__name__=~\"up\"}"
relabel_configs:
- source_labels: [job]
target_label: job
- source_labels: [instance]
target_label: instanceQuery Optimization Tips
- Use
rate()instead of raw counters to reduce data volume - Filter metrics by job or instance to limit scope
- Avoid wildcard matches like
{__name__=~".*"}in federation - Use
match[]to specify only needed metrics
10. Monitoring Federation Performance
| Metric | Description | Unit | Notes |
|---|---|---|---|
| prometheus_remote_storage_samples_total | Total number of samples sent to remote storage | samples | Tracks all samples pushed to remote storage |
| prometheus_remote_storage_samples_failed_total | Total number of samples that failed to send to remote storage | samples | Indicates issues with remote storage connectivity or configuration |
| prometheus_remote_storage_queue_length | Current length of the queue for samples waiting to be sent | samples | High values may indicate remote storage is slow or unreachable |
| prometheus_remote_storage_queue_capacity | Maximum capacity of the queue for samples waiting to be sent | samples | Configurable via remote storage settings |
| prometheus_remote_storage_samples_pending | Number of samples pending to be sent to remote storage | samples | Shows how many samples are queued but not yet sent |
| prometheus_remote_storage_samples_out_of_order_total | Total number of samples rejected due to out-of-order timestamps | samples | May occur if samples are sent with older timestamps than previously received |
Example: Alerting on Remote Storage Failures
groups:
- name: remote-storage-alerts
rules:
- alert: RemoteStorageFailure
expr: rate(prometheus_remote_storage_samples_failed_total[5m]) > 0
for: 10m
labels:
severity: critical
annotations:
summary: "Remote storage is failing to accept samples"
description: "The remote storage backend has failed to accept samples for more than 10 minutes."
Note: Ensure that the remote storage endpoint is configured with sufficient capacity to handle the volume of samples being sent.
11. Implementing Partial Federation
Partial federation allows you to configure a Prometheus instance to scrape only a subset of metrics from another Prometheus instance or federation endpoint, rather than all metrics. This is useful for reducing load and filtering data.
| Configuration Option | Description | Example Value |
|---|---|---|
| scrape_configs | Defines the targets to scrape, including federation endpoints. | [{job_name: 'federate', params: {match[]: ['{job='prometheus'}']}}] |
| match[] | Specifies the metric names to include in federation. | ['{job='prometheus'}'] |
Example: Federation Configuration
scrape_configs:
- job_name: 'federate'
scrape_interval: 15s
honor_labels: true
metrics_path: '/federate'
params:
match[]:
- '{job='prometheus'}'
- '{job='node_exporter'}'
static_configs:
- targets:
- 'prometheus:9090'
Note: The
match[] parameter uses Prometheus's query language syntax to filter metrics. Ensure that the target Prometheus instance supports federation and has the appropriate endpoint exposed.
12. Handling Federation Failures
Example: Configuring Federation with Retry Logic
scrape_configs:
- job_name: "federated-metrics"
scrape_interval: 15s
scrape_timeout: 10s
metrics_path: "/federate"
honor_labels: true
static_configs:
- targets:
- "prometheus-1:9090"
- "prometheus-2:9090"
relabel_configs:
- source_labels: [__address__]
target_label: instance
metric_relabel_configs:
- source_labels: [__name__]
regex: "^go_.*"
action: drop
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| scrape_interval | How often to scrape targets | 15s |
Used for federation scrapes |
| scrape_timeout | Timeout for scrape requests | 10s |
Should be less than scrape_interval |
| metrics_path | Path to scrape metrics from | /federate |
Must match federation endpoint |
| honor_labels | Whether to honor labels from scraped metrics | false |
Recommended for federation to avoid conflicts |
Note: When a federated Prometheus instance is unreachable, it will continue to serve the last known metrics from its local storage.
Warning: Federation can cause cascading failures if not configured properly. Ensure that the upstream Prometheus instances are resilient and have appropriate timeouts.