Skip to main content
Mapagam

Using Prometheus Agent Mode

1. Understanding Agent Mode Purpose

Prometheus Agent Mode is a specialized operational mode designed to reduce the resource overhead and complexity of running Prometheus in large-scale environments. It is particularly useful for scenarios where you want to:

  • Collect metrics from many targets efficiently
  • Reduce the number of active Prometheus instances
  • Centralize metric collection and processing
  • Minimize memory and CPU usage in environments with many targets

In Agent Mode, Prometheus does not scrape targets directly. Instead, it acts as a "pull" agent that receives metrics from other sources (like Prometheus servers or exporters) and forwards them to a central Prometheus instance for storage and querying.

Feature Description Status
Scraping Disabled in Agent Mode DEPRECATED
Remote Write Enabled to send metrics to a central Prometheus instance NEW
Local Storage Disabled or minimal local storage NEW
Configuration Uses a simplified configuration model NEW

Example: Agent Mode Configuration

remote_write:
  - url: http://central-prometheus:9090/api/v1/write
    queue_config:
      capacity: 10000
      max_samples_per_send: 1000
      batch_send_deadline: 5s
Note: Agent Mode is not a replacement for Prometheus in all use cases. It's designed to complement the main Prometheus server by offloading metric collection from many targets.

2. Enabling Agent Mode

Prometheus Agent Mode is a lightweight mode designed for collecting metrics from targets and forwarding them to a remote storage system. It's particularly useful in environments where you want to reduce the resource footprint of Prometheus while still maintaining metric collection capabilities.

Configuration Option Description Default Value Notes
agent.mode Enables or disables agent mode false Set to true to enable agent mode
agent.remoteWrite.url Remote URL to send metrics to None Required when agent mode is enabled
agent.remoteWrite.basicAuth.username Username for basic auth None Optional, if authentication is required
agent.remoteWrite.basicAuth.password Password for basic auth None Optional, if authentication is required
agent.remoteWrite.writeRelabelConfigs Relabeling rules for metrics before sending None Optional, allows filtering or renaming metrics

Example: Enabling Agent Mode in Prometheus Configuration

agent:
  mode: true
  remoteWrite:
    url: https://prometheus.example.com/api/v1/write
    basicAuth:
      username: agent-user
      password: agent-password
Note: Agent mode does not support scraping targets directly. It only forwards metrics to a remote endpoint.

3. Configuring Remote Write Endpoint

Option Values Description
remote_write.url http://remote-prometheus:9090/api/v1/write URL of the remote write endpoint to send samples to.
remote_write.name remote Name of the remote write configuration block.
remote_write.basic_auth.username user Username for basic authentication.
remote_write.basic_auth.password password Password for basic authentication.
remote_write.proxy_url http://proxy:3128 Proxy URL to use for remote write requests.
remote_write.timeout 30s Timeout for remote write requests.
remote_write.write_retries 3 Number of retries for failed writes.

Example: Configuring remote_write in Prometheus

remote_write:
  - name: "remote"
    url: http://remote-prometheus:9090/api/v1/write
    basic_auth:
      username: "user"
      password: "password"
    timeout: 30s
    write_retries: 3

4. Understanding Storage Differences

Storage Type Description Key Characteristics Use Case
Remote Write Writes metrics to a remote endpoint High throughput, scalable, supports compression Centralized monitoring, long-term storage
Local Storage Stores metrics on the local filesystem Low latency, simple setup, limited by disk space Short-term metrics, testing, isolated environments
In-memory Stores metrics in memory only Fast access, no persistence, volatile Temporary metrics, real-time dashboards

Example: Remote Write Configuration

remote_write:
  - url: http://prometheus-remote.example.com/api/v1/write
    queue_config:
      capacity: 1000
      max_shards: 10
Note: Remote write is ideal for large-scale deployments where centralized storage and long-term retention are required.

5. Using Agent for Edge Deployments

Prometheus Agent Mode is particularly useful in edge deployments where resources are limited and network connectivity may be intermittent. The agent mode allows for efficient collection and forwarding of metrics to a central Prometheus server.

Feature Description Notes
agent.mode true Enables agent mode in Prometheus configuration
agent.remoteWrite.url http://central-prometheus:9090/api/v1/write URL of the central Prometheus server to forward metrics
agent.remoteWrite.queueConfig {} Configuration for the queue used to buffer metrics before sending

Example: Agent Configuration for Edge Deployment

agent:
  mode: true
  remoteWrite:
    url: http://central-prometheus:9090/api/v1/write
    queueConfig:
      capacity: 10000
      maxRetries: 3
Note: In edge deployments, it's crucial to configure the agent with appropriate queue sizes and retry mechanisms to handle intermittent network connectivity.

Key Considerations for Edge Deployments

  • Resource constraints: Agent mode is optimized to reduce memory and CPU usage.
  • Network reliability: The agent can buffer metrics locally if the central server is unreachable.
  • Scalability: Multiple agents can forward metrics to a single central Prometheus server.

6. Configuring WAL in Agent Mode

Configuration Option Description Default Value Notes
agent.wal-compression Enables WAL compression to reduce disk space usage. false Set to true to enable compression.
agent.wal-dir Directory where the WAL files are stored. /data/wal Must be writable by the Prometheus process.
agent.wal-truncate-frequency How often the WAL is truncated to remove old data. 15m Adjust based on retention needs.
agent.wal-segment-size Size of each WAL segment file. 128MB Can be adjusted to optimize I/O performance.

Example: WAL Configuration in Agent Mode

agent:
  wal-compression: true
  wal-dir: /prometheus/wal
  wal-truncate-frequency: 30m
  wal-segment-size: 256MB
Note: WAL configuration in agent mode is optimized for low memory usage and efficient disk I/O. Ensure sufficient disk space for WAL files.

7. Implementing Agent Discovery

Prometheus Agent Mode supports service discovery through various mechanisms to dynamically discover and monitor targets. These configurations are typically defined in the agent's configuration file.

Discovery Type Configuration Description
static_config { targets: ["target1", "target2"] } Defines a static list of targets to monitor. Useful for small, fixed sets of endpoints.
file_sd_config { files: ["targets.json"] } Discovers targets from a JSON or YAML file. The file must contain an array of target groups.
consul_sd_config { server: "localhost:8500", token: "token" } Discovers targets from Consul. Requires Consul agent running and configured.
kubernetes_sd_config { role: "pod", namespace: "default" } Discovers targets from Kubernetes pods or services. Requires Kubernetes API access.
ec2_sd_config { region: "us-east-1", filters: [{ name: "tag:Environment", values: ["prod"] }] } Discovers targets from AWS EC2 instances. Requires AWS credentials and permissions.

Example: Kubernetes Service Discovery

scrape_configs:
  - job_name: "agent-k8s"
    kubernetes_sd_config:
      role: pod
      namespace:
        name: "default"
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
Note: The agent mode does not support all service discovery mechanisms available in the Prometheus server. Ensure that the selected SD mechanism is compatible with the agent mode.

8. Setting Resource Limits

Resource Type Default Limit Configuration Key Notes
Memory 512Mi --agent.memory-limit Maximum memory usage for the agent process
CPU 500m --agent.cpu-limit Maximum CPU usage for the agent process
Storage 1Gi --agent.storage-limit Maximum disk space used for local storage

Example: Configuring Resource Limits

agent:
  memoryLimit: "1Gi"
  cpuLimit: "1"
  storageLimit: "2Gi"
Note: Resource limits should be set based on the expected load and data volume in your environment.

9. Monitoring Agent Health

Example: Agent Health Check Endpoint

# Prometheus Agent mode health check
- name: prometheus-agent-health
  http_get:
    url: http://localhost:9093/-/healthy
    timeout: 5s
Endpoint Description Status Code Notes
/-/healthy Reports if the agent is healthy 200 OK Used for liveness probes
/-/ready Reports if the agent is ready to serve requests 200 OK Used for readiness probes
Note: The /-/healthy and /-/ready endpoints are part of the Prometheus Agent mode's built-in monitoring capabilities.

Health Check Configuration

  • Default port for health checks: 9093
  • Endpoints are accessible at http://localhost:9093/-/healthy
  • Can be used with Kubernetes liveness and readiness probes

10. Using Agent with Thanos Receive

When using Prometheus in Agent mode with Thanos Receive, the agent acts as a proxy for metrics ingestion and forwarding to Thanos components.

Configuration Option Description Default Value Notes
agent.remoteWrite.url URL to send samples to None Required when using Thanos Receive
agent.remoteWrite.basicAuth.username Username for basic auth None Optional, if authentication is required
agent.remoteWrite.basicAuth.password Password for basic auth None Optional, if authentication is required
agent.remoteWrite.timeout Timeout for remote write requests 30s Can be adjusted based on network conditions

Example: Agent Configuration with Thanos Receive

agent:
  remoteWrite:
    url: http://thanos-receive.example.com/api/v1/write
    basicAuth:
      username: prometheus-agent
      password: "secret"
    timeout: 30s
Note: Ensure that the Thanos Receive endpoint is accessible from the Prometheus Agent and that appropriate network policies are in place.

11. Implementing Agent High Availability

Example: Configuring Prometheus Agent for High Availability

# prometheus-agent.yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "node"
    static_configs:
      - targets: ["localhost:9100"]

agent:
  mode: "agent"
  remote_write:
    - url: "http://prometheus-server-1:9090/api/v1/write"
      queue_config:
        capacity: 10000
        max_samples_per_send: 1000
      headers:
        Authorization: "Bearer secret-token"
Configuration Description Values Notes
agent.mode Enables agent mode "agent" Required for agent mode
remote_write.url URL to remote write endpoint "http://prometheus-server-1:9090/api/v1/write" Must be accessible from agent
remote_write.queue_config.capacity Max samples in queue 10000 Prevents memory overflow
remote_write.queue_config.max_samples_per_send Max samples per send 1000 Optimizes batch size
Note: Agent mode requires a remote write endpoint to store metrics.
Warning: Ensure all agents point to the same remote write endpoint for consistency.

High Availability Setup Steps

  • Deploy multiple Prometheus agents in different availability zones
  • Configure each agent to remote write to the same Prometheus server or cluster
  • Use consistent configuration across all agents
  • Monitor agent health and remote write status

12. Debugging Agent Issues

Issue Description Solution
Agent not scraping targets Targets are not being scraped by the Prometheus Agent Check agent configuration for target discovery settings and verify that scrape_configs are correctly defined.
High memory usage Agent consumes more memory than expected Review scrape intervals and retention settings. Consider reducing the number of metrics or increasing memory limits.
Missing metrics in remote storage Metrics are not reaching remote storage systems (e.g., Thanos, Cortex) Verify remote write configuration and check for network issues or authentication problems.
Agent fails to start Agent fails to boot up with an error Check logs for startup errors and ensure all required configuration files are valid and accessible.

Example: Agent logs for debugging

prometheus --config.file=/etc/prometheus/prometheus.yml \
  --agent.mode=true \
  --log.level=debug
Note: Use --log.level=debug to get more verbose logs for troubleshooting.
Warning: Ensure that the agent has proper permissions to access target endpoints and read configuration files.

Was this article helpful?

Related Cheat Sheets