Skip to main content
Mapagam

Implementing Remote Read

1. Configuring Remote Read

Configuration Option Description Example Value Notes
remote_read Defines remote read configurations for querying data from other Prometheus instances or compatible systems. 
[{
  url: "http://remote-prometheus.example.com/api/v1/query",
  name: "remote-instance",
  remote_timeout: "30s",
  read_recent: true,
  headers: {
    "Authorization": "Bearer token123"
  }
}]
 Must be an array of remote read configurations.

Example: Basic Remote Read Configuration

remote_read:
  - url: "http://remote-prometheus.example.com/api/v1/query"
    name: "remote-instance"
    remote_timeout: "30s"

Example: Remote Read with Authentication

remote_read:
  - url: "http://remote-prometheus.example.com/api/v1/query"
    name: "secure-instance"
    remote_timeout: "30s"
    headers:
      Authorization: Bearer <token>
Note: Ensure that the remote Prometheus instance allows queries from the local instance to avoid permission errors.
Warning: Setting read_recent to true can cause performance issues if the remote system is under heavy load.

2. Setting Remote Read URL

Property Value Description
remote_read.url http://remote-prometheus:9090/api/v1/read URL of the remote Prometheus instance to read from
remote_read.name remote-read Name of the remote read configuration
remote_read.remote_timeout 30s Timeout for remote read requests

Example: Configuring remote read URL in Prometheus config

remote_read:
  - url: http://remote-prometheus:9090/api/v1/read
    name: remote-read
    remote_timeout: 30s

3. Configuring Read Recent

Option Values Description
remote_read [] List of remote read configurations
remote_read.name string Name of the remote read configuration
remote_read.url string URL of the remote Prometheus instance to read from
remote_read.read_recent bool Whether to read recent data from the remote instance
remote_read.remote_timeout duration Timeout for requests to the remote instance
remote_read.basic_auth object Basic authentication credentials for the remote instance
remote_read.bearer_token string Bearer token for authentication with the remote instance
remote_read.bearer_token_file string Path to a file containing a bearer token for the remote instance
remote_read.proxy_url string Proxy URL to use for requests to the remote instance
remote_read.headers map[string]string Additional headers to send with requests to the remote instance
remote_read.read_recent bool Whether to read recent data from the remote instance

Example: Configuring Read Recent

remote_read:
  - name: "remote-instance"
    url: "https://prometheus.example.com/api/v1/read"
    read_recent: true
    remote_timeout: 30s
    basic_auth:
      username: "user"
      password: "password"

4. Using Remote Timeout

Configuration Values Description
remote_read.timeout 30s Timeout for remote read requests to the Prometheus server
remote_read.max_bytes_in_frame 8192 Maximum number of bytes to read per frame in a remote read response

Example: Configuring remote read timeout

remote_read:
  - url: "http://prometheus-remote-read:9090/api/v1/read"
    timeout: 30s
    max_bytes_in_frame: 8192
Note: Setting timeout too low may cause read failures during high load or large query responses.

5. Implementing Basic Auth

Configuration Option Description Values Notes
remote_read.basic_auth.username Username for basic authentication string Required if basic auth is enabled
remote_read.basic_auth.password Password for basic authentication string Must be provided as a secret
remote_read.basic_auth.password_file Path to file containing password string Alternative to inline password, preferred for security

Example: Basic Auth Configuration

remote_read:
  - url: http://remote-prometheus.example.com/api/v1/read
    basic_auth:
      username: "prometheus"
      password: "secret123"
Note: It is recommended to use password_file instead of inline password for better security practices.

6. Configuring TLS

Field Description Example
tls_config Configures TLS settings for remote read connections. { "ca_file": "/path/to/ca.crt", "cert_file": "/path/to/client.crt", "key_file": "/path/to/client.key" }

Example: TLS Configuration in Prometheus Config

remote_read:
  - url: https://prometheus-remote.example.com/read
    tls_config:
      ca_file: /etc/ssl/certs/ca.crt
      cert_file: /etc/ssl/certs/client.crt
      key_file: /etc/ssl/private/client.key

7. Using Required Matchers

When querying Prometheus for remote read, you can define required matchers to filter metrics at the remote storage level. These matchers ensure that only data matching specific label criteria is returned.

Matcher Description Example
match[] Defines a label matcher to filter metrics by label name and value. match[]={job="prometheus"}
match[] Supports regex matching using =~. match[]={job=~"prometheus|alertmanager"}

Example: Using Required Matchers in Remote Read

remote_read:
  - url: "http://remote-prometheus:9090/api/v1/read"
    required_matchers:
      match[]: "job=prometheus"
Note: Required matchers are applied at the remote read endpoint and can significantly reduce data transfer if used correctly.

8. Implementing Filter External Labels

Property Description Values Notes
external_labels Labels to be added to every metric {"job": "prometheus", "env": "prod"} Used to filter or tag metrics
label_name Name of the label to filter on job Must match existing label in metrics
label_value Value to filter by prometheus Matches metrics with this label value

Example: Filtering External Labels

remote_read:
  - url: "http://prometheus.example.com/api/v1/read"
    filter_external_labels:
      job: prometheus
      env: prod
Note: The filter_external_labels option is used to apply additional labels to metrics before they are sent to the remote read endpoint.

9. Configuring Name Filtering

Option Values Description
remote_read.name_filter string Specifies the name filter to apply when reading from remote Prometheus instances.
remote_read.name_filter.exclude boolean If true, excludes metrics that match the name filter pattern. If false, includes only those matching the pattern.
remote_read.name_filter.pattern string The regex pattern used to match metric names for filtering.

Example: Name Filtering Configuration

remote_read:
  - url: "http://prometheus-remote.example.com/api/v1/read"
    name_filter:
      pattern: "^metrics_.*"
      exclude: false
Note: The name_filter option is available in Prometheus v2.35.0 and later.

10. Monitoring Remote Read

Property Description Default Value Notes
remote_read.url URL of the remote read endpoint None Required field for remote read configuration
remote_read.name Name of the remote read configuration None Used for identifying the remote read instance in logs and metrics
remote_read.remote_timeout Timeout for requests to the remote read endpoint 30s Configurable to handle slow or unresponsive remote systems
remote_read.read_recent Whether to read recent data from the remote endpoint true Enables reading of recent time series data from remote storage
remote_read.basic_auth Basic authentication credentials for the remote read endpoint None Supports username/password authentication
remote_read.bearer_token Bearer token for the remote read endpoint None Supports token-based authentication for secure endpoints

Example: Remote Read Configuration

remote_read:
  - url: "https://remote-prometheus.example.com/api/v1/read"
    name: "remote-read-instance"
    remote_timeout: 1m
    read_recent: true
    basic_auth:
      username: "user"
      password: "password"
Note: Ensure that the remote read endpoint supports the Prometheus remote read protocol and is configured to handle queries from your local Prometheus instance.

11. Optimizing Remote Read Performance

Configuration Description Default Value Notes
remote_read Defines remote read configurations None Array of remote read configs
url URL of the remote Prometheus instance to read from None Required field
name Unique name for the remote read configuration None Used for labeling in metrics
remote_timeout Timeout for requests to the remote read endpoint 30s Must be greater than 0
read_recent Whether to read recent samples from the remote storage true Improves query performance
bearer_token Bearer token for authentication None Alternative to basic auth
basic_auth Basic authentication credentials None Username and password

Example: Remote Read Configuration

remote_read:
  - url: "https://prometheus.example.com/api/v1/read"
    name: "remote-prometheus"
    remote_timeout: 1m
    read_recent: true
    basic_auth:
      username: "user"
      password: "password"
Note: Setting read_recent: false can reduce latency but may cause data inconsistency.
Performance Metric Description Optimization Strategy
Query Latency Time taken to fetch data from remote storage Reduce remote_timeout, use efficient query ranges
Memory Usage Amount of memory used for remote read operations Limit concurrent reads, use max_samples
Network Bandwidth Data transferred over network for remote reads Compress data with gzip, limit query size

Optimization Tips

  • Set remote_timeout to a value that balances responsiveness and reliability
  • Use read_recent: true to reduce query load on remote systems
  • Implement authentication using bearer_token or basic_auth for security
  • Monitor query latency and adjust max_samples to prevent memory exhaustion

12. Combining Local and Remote Data

When implementing remote read in Prometheus, you can combine local and remote data to create a unified view of your metrics. This is particularly useful for long-term storage and querying.

Configuration Option Description Values Notes
remote_read Defines remote read configurations Array of remote read configs Each config specifies a remote read endpoint
url Remote read endpoint URL String (URL) Must be accessible from Prometheus server
remote_timeout Timeout for remote read requests Duration (e.g., 30s) Default is 30s
read_recent Whether to read recent data from remote storage Boolean Defaults to true
name Identifier for the remote read configuration String Used in metrics and logs

Example: Combining Local and Remote Data

remote_read:
  - url: "https://prometheus-remote.example.com/read"
    name: "remote-store"
    read_recent: true
    remote_timeout: 1m
Note: When combining local and remote data, Prometheus will merge results from both sources. The query engine ensures that local and remote metrics are merged correctly, maintaining consistency in the results.

Key Considerations

  • Remote read configurations must be defined in the Prometheus configuration file.
  • Ensure that remote endpoints are accessible and secure to prevent data leakage.
  • Use read_recent to control whether recent local data is included in queries.
  • Set appropriate remote_timeout values to avoid query timeouts during high load.

Was this article helpful?

Related Cheat Sheets