Implementing Remote Write

1. Configuring Remote Write

Option	Values	Description
remote_write.url	`http://remote-prometheus:9090/api/v1/write`	URL of the remote write endpoint to send samples to.
remote_write.name	`remote-write-1`	Name of the remote write configuration.
remote_write.basic_auth.username	`user`	Username for basic authentication.
remote_write.basic_auth.password	`password`	Password for basic authentication.
remote_write.write_retries	`3`	Number of retries to attempt on write failures.
remote_write.queue_config.capacity	`1000`	Maximum number of samples to queue before dropping.
remote_write.queue_config.batch_send_deadline	`5s`	Maximum time to wait before sending a batch.
remote_write.queue_config.max_samples_per_send	`1000`	Maximum number of samples to send in one batch.

Example: Basic remote write configuration

remote_write:
  - url: http://remote-prometheus:9090/api/v1/write
    name: remote-write-1
    basic_auth:
      username: user
      password: password
    queue_config:
      capacity: 1000
      batch_send_deadline: 5s
      max_samples_per_send: 1000

2. Setting Remote Write URL

Property	Value	Description
remote_write.url	`http://remote-prometheus:9090/api/v1/write`	URL to send samples to for remote storage
remote_write.name	`remote-storage`	Name of the remote write configuration

Example: Configure remote write URL

remote_write:
  - url: http://remote-prometheus:9090/api/v1/write
    name: remote-storage

3. Configuring Write Relabeling

Field	Description	Example
action	Relabel action to perform. Valid values: `replace`, `keep`, `drop`, `hashmod`, `labelmap`, `labeldrop`, `labelkeep`.	`replace`
source_labels	List of label names to read values from.	`[__name__]`
separator	Separator used to join source label values.	`;`
target_label	Label name to write the result to.	`job`
regex	Regular expression to match source label values.	`^.*$`
replacement	Replacement value for matched regex groups.	`prometheus`

Example: Relabeling for Job Name

remote_write:
  - url: http://prometheus.example.com/api/v1/write
    write_relabel_configs:
      - action: replace
        source_labels: [__name__]
        target_label: job
        replacement: prometheus

Note: Write relabeling occurs before metrics are sent to the remote endpoint, allowing for filtering and transformation of metric labels.

4. Setting Queue Configuration

Property	Description	Default Value	Notes
queue_config.capacity	Maximum number of samples that can be queued	`10000`	Adjust based on memory constraints and ingestion rate
queue_config.batch_send_deadline	Maximum time to wait before sending a batch	`5s`	Higher values can increase latency but reduce network overhead
queue_config.max_samples_per_send	Maximum number of samples to send in a single batch	`1000`	Can be tuned to balance between latency and throughput
queue_config.max_retries	Maximum number of retries for failed sends	`10`	Set to 0 to disable retries
queue_config.retry_interval	Initial retry interval for failed sends	`1s`	Exponential backoff is applied after this interval
queue_config.max_backoff_interval	Maximum retry interval for failed sends	`10s`	Prevents excessive delays in case of persistent failures

Example: Remote Write Queue Configuration

remote_write:
  - url: http://prometheus-remote-write:9090/api/v1/write
    queue_config:
      capacity: 50000
      batch_send_deadline: 10s
      max_samples_per_send: 2000
      max_retries: 5
      retry_interval: 2s
      max_backoff_interval: 30s

Note: Adjust queue configuration parameters based on your system's capacity and desired performance characteristics. Higher capacity values may increase memory usage, while longer batch_send_deadline values can reduce network overhead but increase latency.

5. Implementing Basic Auth

Configuration Option	Description	Value
remote_write.basic_auth.username	Username for basic authentication	`prometheus`
remote_write.basic_auth.password	Password for basic authentication	`secret`

Example: Configuring Basic Auth in Prometheus

remote_write:
  - url: http://remote-prometheus:9091/api/v1/write
    basic_auth:
      username: prometheus
      password: secret

6. Configuring TLS

Field	Description	Example
tls_config	Configures TLS settings for the remote write endpoint.	`{ "ca_file": "/path/to/ca.crt", "cert_file": "/path/to/client.crt", "key_file": "/path/to/client.key" }`

Example: TLS Configuration in Prometheus Config

remote_write:
  - url: https://prometheus-remote.example.com/api/v1/write
    tls_config:
      ca_file: /etc/ssl/certs/ca.crt
      cert_file: /etc/ssl/certs/client.crt
      key_file: /etc/ssl/private/client.key
      server_name: prometheus-remote.example.com

Note: Ensure that the CA certificate, client certificate, and key files are accessible to the Prometheus process and have appropriate permissions.

7. Using External Labels

Label Name	Description	Usage
job	Identifies the job that is being scraped	Used to group metrics by scraping job
instance	Identifies the instance that is being scraped	Uniquely identifies a target within a job
cluster	Identifies the cluster to which a target belongs	Used for multi-cluster setups
environment	Identifies the environment (e.g., dev, staging, prod)	Used for filtering and routing metrics

Example: Configuring External Labels in Prometheus

global:
  external_labels:
    cluster: "prod-cluster"
    environment: "production"

remote_write:
  - url: "http://remote-prometheus:9090/api/v1/write"
    write_relabel_configs:
      - source_labels: [job]
        target_label: job
        regex: ".*"
        replacement: "${job}-prod"

Note: External labels are applied to all samples before they are sent to the remote storage. They can be overridden by relabeling rules.

8. Setting Capacity and Batching

Option	Values	Description
remote.write.capacity	`100`	Maximum number of samples that can be queued for remote write before blocking.
remote.write.batch_send_deadline	`5s`	Maximum time to wait before sending a batch of samples.
remote.write.max_samples_per_send	`1000`	Maximum number of samples to send in a single batch.
remote.write.max_backoff	`10s`	Maximum backoff time between retries.
remote.write.min_backoff	`30ms`	Minimum backoff time between retries.

Example: Configuring remote write batching

remote_write:
  - url: http://prometheus-remote-write:9090/api/v1/write
    capacity: 200
    batch_send_deadline: 10s
    max_samples_per_send: 2000
    queue_config:
      capacity: 200
      max_samples_per_send: 2000
      batch_send_deadline: 10s

Note: Adjusting capacity and batch_send_deadline can help balance between memory usage and write efficiency.

9. Configuring Send Exemplars

Option	Values	Description
send_exemplars	`true` \| `false`	Enable or disable sending exemplars to the remote write endpoint. Defaults to `false`.

Example: Enabling Send Exemplars

remote_write:
  - url: http://remote-prometheus:9090/api/v1/write
    send_exemplars: true

Note: Exemplars are only sent if the remote endpoint supports them. Ensure that your remote storage system (e.g., Cortex, Thanos) is configured to accept and store exemplars.

10. Implementing Retry Logic

Configuration Option	Description	Default Value	Notes
remote_write.retry	Enables or disables retry logic for remote write requests.	`false`	Set to `true` to enable retry logic.
remote_write.retry_interval	Time between retry attempts (in seconds).	`5s`	Adjust based on network stability and desired resilience.
remote_write.max_retries	Maximum number of retry attempts before giving up.	`10`	Set to `-1` for unlimited retries.

Example: Configuring Retry Logic

remote_write:
  - url: http://prometheus-remote.example.com/api/v1/write
    retry: true
    retry_interval: 10s
    max_retries: 5

Note: Retry logic is particularly useful in environments with intermittent network issues or when writing to remote storage systems that may temporarily be unavailable.

11. Monitoring Remote Write

Property	Description	Values	Notes
remote_write.url	Remote URL to send samples to	`http://remote-prometheus:9090/api/v1/write`	Required for remote write
remote_write.name	Remote write name for identification	`remote-write-1`	Used in metrics and logs
remote_write.write_retries	Number of retries on write failure	`3`	Default value
remote_write.batch_send_deadline	Max time to wait before sending a batch	`5s`	Default value
remote_write.send_exemplars	Send exemplars with samples	`true`	Enabled by default

Example: Configuring Remote Write

remote_write:
  - url: http://remote-prometheus:9090/api/v1/write
    name: remote-write-1
    write_retries: 3
    batch_send_deadline: 5s
    send_exemplars: true

Note: Ensure the remote Prometheus instance is configured to accept samples from the sending instance.

Remote Write Monitoring Metrics

prometheus_remote_storage_samples_total - Total samples sent
prometheus_remote_storage_samples_failed_total - Failed samples count
prometheus_remote_storage_queue_length - Current queue size
prometheus_remote_storage_max_samples_per_send - Max samples per send

12. Tuning Remote Write Performance

Configuration Option	Description	Default Value	Notes
remote_write.queue_config.capacity	Maximum number of samples that can be queued for sending.	`10000`	Higher values can improve throughput but increase memory usage.
remote_write.queue_config.max_samples_per_send	Maximum number of samples to send in a single batch.	`1000`	Increasing this can reduce the number of HTTP requests but may increase latency.
remote_write.queue_config.batch_send_deadline	Maximum time to wait before sending a batch even if it's not full.	`5s`	Adjust to balance between latency and throughput.
remote_write.queue_config.max_retries	Maximum number of retries for failed writes.	`10`	Higher values increase resilience but may cause more backpressure.
remote_write.queue_config.retry_interval	Initial interval between retries (doubles each time).	`500ms`	Adjust based on expected network stability and server response times.

Example: Tuning remote_write settings in Prometheus config

remote_write:
  - url: http://prometheus-remote.example.com/api/v1/write
    queue_config:
      capacity: 20000
      max_samples_per_send: 2000
      batch_send_deadline: 10s
      max_retries: 15
      retry_interval: 1s

Note: Adjusting capacity and max_samples_per_send can significantly impact performance depending on your remote storage's capacity and network conditions.

Performance Tuning Tips

Increase capacity to handle bursts in metric volume.
Adjust max_samples_per_send to balance between batch size and latency.
Use batch_send_deadline to ensure timely delivery under low load.
Monitor retry behavior and adjust max_retries to prevent overwhelming the remote system.