Skip to main content
Mapagam

Working with Exporters

1. Understanding Exporter Pattern

Aspect Description Usage
Purpose Collects metrics from applications and exposes them in Prometheus format Used to instrument services or systems for monitoring
Metrics Format text/plain format with # HELP and # TYPE comments Standard Prometheus exposition format
Endpoint /metrics Default endpoint where Prometheus scrapes metrics

Example: Exporter Configuration

scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['localhost:9090']
Note: Exporters are essential for integrating third-party systems or services that do not natively expose metrics in Prometheus format.

2. Using Node Exporter

Configuration Option Description Default Value Notes
--collector.disable-defaults Disables collection of default collectors false Set to true to disable all default collectors
--collector.filesystem.ignored-fs-types Filesystem types to ignore ^/(sys|proc|dev|run)/ Regex pattern to exclude filesystems
--collector.netdev.ignored-devices Network devices to ignore ^$ Regex pattern to exclude network devices

Example: Starting Node Exporter with Custom Configuration

# Start Node Exporter with custom filesystem and network device filters
node_exporter --collector.filesystem.ignored-fs-types="^/(sys|proc|dev)" \
              --collector.netdev.ignored-devices="^lo$"
Note: Node Exporter requires root privileges to collect some system metrics like disk I/O stats.

Key Features of Node Exporter

  • Collects hardware and OS metrics from Linux systems
  • Exposes metrics in Prometheus format via HTTP endpoint
  • Supports various collectors for different system components
  • Can be configured to ignore specific filesystems or network devices

3. Using Blackbox Exporter

The Blackbox Exporter allows you to monitor the availability and latency of external services by probing them using various protocols such as HTTP, HTTPS, DNS, TCP, and ICMP.

Configuration Option Description Default Value
module Defines the probe module to use. Modules include http_2xx, tcp_connect, dns_query, etc. http_2xx
target The target URL or address to probe. For example, https://example.com or 8.8.8.8:53. None
timeout Timeout for the probe in seconds. Default is 10s. 10s

Example: Configuring a Blackbox Exporter job

scrape_configs:
  - job_name: "blackbox"
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://prometheus.io
          - https://grafana.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

Example: Probe configuration for DNS queries

modules:
  dns_query:
    prober: dns
    dns:
      query_name: "prometheus.io"
      query_type: "A"
      timeout: 5s
Note: Ensure that the Blackbox Exporter is running and accessible from your Prometheus instance to successfully scrape metrics.

4. Integrating Pushgateway

The Pushgateway is a Prometheus component that allows you to expose metrics from jobs that are short-lived or run on demand. It acts as an intermediary between the job and Prometheus, collecting metrics from jobs that cannot be scraped directly.

Configuration Description Default Value
pushgateway.address Address of the Pushgateway server http://localhost:9091
pushgateway.job.name Name of the job to push metrics for default-job
pushgateway.push.interval Interval to push metrics (in seconds) 60

Example: Pushing Metrics to Pushgateway

const client = require('prom-client');
const pushgateway = new client.Pushgateway('http://localhost:9091');

const counter = new client.Counter({
  name: 'my_counter',
  help: 'A simple counter',
});

counter.inc();

pushgateway.pushAdd({ jobName: 'my-job' }, (err) => {
  if (err) {
    console.error('Failed to push metrics:', err);
  } else {
    console.log('Metrics pushed successfully');
  }
});
Note: Pushgateway is not suitable for high-volume or frequent metric updates. It's best used for batch jobs or short-lived processes.
Warning: Metrics pushed to Pushgateway are not automatically removed. You must explicitly delete them or set a TTL (time-to-live) to prevent stale metrics from accumulating.

5. Configuring Custom Exporters

Custom exporters allow you to expose metrics from applications or systems that are not natively supported by Prometheus. These exporters can be written in any language and must expose metrics in the Prometheus text format.

Configuration Option Description Example Value
exporter.binary Path to the exporter binary executable /usr/local/bin/my-exporter
exporter.args Arguments to pass to the exporter binary --web.listen-address=:9100
exporter.scrape_interval How often to scrape metrics from the exporter 30s
exporter.timeout Timeout for scraping metrics from the exporter 10s

Example: Running a Custom Exporter

# Start the custom exporter
/usr/local/bin/my-exporter --web.listen-address=:9100 & 

# Add to Prometheus configuration
scrape_configs:
  - job_name: 'custom-exporter'
    static_configs:
      - targets: ['localhost:9100']
Note: Ensure the exporter binary is executable and accessible to the Prometheus process.
Warning: Custom exporters must expose metrics in the Prometheus text format to be scraped correctly.

6. Setting Exporter-Specific Parameters

Parameter Description Example Value Notes
collector.disable Disables the collector module false Set to true to disable
collector.timeout Timeout for collector requests 30s Must be a valid duration string
collector.config.path Path to the collector config file /etc/collector/config.yaml Must be a valid file path

Example: Configuring Exporter Parameters

collector:
  disable: false
  timeout: 30s
  config:
    path: /etc/collector/config.yaml

7. Using Exporter Labels

Label Name Description Usage Example Notes
job Identifies the job that is being scraped "job": "node_exporter" Required for all metrics
instance Identifies the instance being scraped "instance": "localhost:9100" Helps distinguish between multiple instances
exporter Identifies the specific exporter used "exporter": "node_exporter" Useful for multi-exporter setups
environment Environment tag for grouping metrics "environment": "production" Used for filtering and alerting

Example: Adding Labels to Metrics

node_cpu_seconds_total{job="node_exporter", instance="localhost:9100", mode="idle"} 1234.56
node_cpu_seconds_total{job="node_exporter", instance="localhost:9100", mode="system"} 456.78
Note: Exporter labels are automatically added by Prometheus when scraping metrics. These labels help in identifying the source of metrics and can be used for filtering and alerting.
Warning: Avoid using too many labels as they can increase memory usage and reduce query performance.

8. Implementing Textfile Collector

The Textfile Collector is a Prometheus exporter that allows you to expose custom metrics from files on disk. It reads text files and converts their content into Prometheus metrics.

Configuration Option Description Default Value
--collector.textfile.directory Directory to read text files from /var/lib/prometheus/textfile_collector
--collector.textfile.skip-atomic Whether to skip atomic file writes false

Example: Textfile Collector Configuration

collector:
  textfile:
    directory: /var/lib/prometheus/textfile_collector
    skip-atomic: false

Textfile Collector supports the following metric formats:

  • # TYPE - Defines the metric type (counter, gauge, histogram, etc.)
  • # HELP - Provides a description of the metric
  • # UNIT - Defines the unit of the metric (optional)

Example: Textfile Format

# TYPE my_metric gauge
# HELP my_metric A sample gauge metric
my_metric 42.0

# TYPE request_count counter
# HELP request_count Number of HTTP requests
request_count{method="GET"} 100.0
request_count{method="POST"} 50.0

Files must end with .prom to be processed by the Textfile Collector.

Note: The collector reads files atomically. If a file is written to in a non-atomic way, it may be read while incomplete, leading to errors.
Warning: Files must be written to a temporary location and then moved or renamed to their final name to ensure atomicity.

9. Monitoring Exporter Health

Property Description Default Value Notes
health_status Indicates the current health status of the exporter healthy Values: healthy, unhealthy, degraded
last_updated Timestamp of the last health check null Unix timestamp in seconds
exporter_up Boolean indicating if the exporter is up and running false Used in Prometheus to track exporter availability

Example: Health Check Endpoint

GET /-/healthy

Example: Health Status Response

{
  "status": "healthy",
  "last_updated": 1678886400,
  "exporter_up": true
}
Note: The /-/healthy endpoint is a standard Prometheus exporter health check and should return a 200 OK status when the exporter is healthy.

10. Handling Exporter Timeouts

Configuration Description Default Value Notes
scrape.timeout Timeout for scraping metrics from exporters 10s Can be overridden per job
scrape.scrape-timeout Alias for scrape.timeout 10s Deprecated in favor of scrape.timeout

Example: Configuring Scrape Timeout

scrape_configs:
  - job_name: 'exporter-job'
    static_configs:
      - targets: ['localhost:9100']
    scrape_timeout: 15s
Note: If an exporter does not respond within the specified timeout, Prometheus marks the scrape as failed and logs a warning.

Timeout Handling Best Practices

  • Set scrape timeouts based on exporter response times
  • Use scrape_timeout in job-level configuration to override global defaults
  • Monitor for frequent timeout errors as potential performance issues

11. Using Community Exporters

Exporter Description Metrics Provided Installation
node_exporter Exports metrics from the Node.js runtime and system-level information. node_cpu_seconds_total, node_memory_bytes, node_filesystem_size_bytes 
curl -O https://github.com/prometheus/node_exporter/releases/latest/download/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvfz node_exporter-1.7.0.linux-amd64.tar.gz
./node_exporter &
mysql_exporter Exports metrics from MySQL databases. mysql_up, mysql_global_status_threads_connected, mysql_info_schema_tables 
docker run -d --name mysql-exporter \
  -p 9104:9104 \
  prom/mysqld-exporter
nginx_exporter Exports metrics from NGINX web servers. nginx_connections_active, nginx_http_requests_total, nginx_up 
docker run -d --name nginx-exporter \
  -p 9113:9113 \
  quay.io/prometheus/nginx-exporter
redis_exporter Exports metrics from Redis instances. redis_up, redis_memory_used_bytes, redis_commands_processed_total 
docker run -d --name redis-exporter \
  -p 9121:9121 \
  oliver006/redis_exporter
kafka_exporter Exports metrics from Apache Kafka clusters. kafka_broker_info, kafka_topic_partition_count, kafka_consumer_group_lag 
docker run -d --name kafka-exporter \
  -p 9308:9308 \
  quay.io/prometheus/kafka-exporter

Example: Configuring Prometheus to Scrape Node Exporter

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
Note: Ensure that the exporter is running and accessible from the Prometheus server before adding it to the configuration.

12. Implementing Multi-Target Exporters

Multi-target exporters allow scraping metrics from multiple targets using a single exporter instance. This is useful for reducing resource usage and simplifying configuration.

Configuration Description Example
scrape_configs Defines the targets to scrape and their configurations. {"job_name": "node_exporter", "static_configs": [{"targets": ["localhost:9100", "server2:9100"]}]}
relabel_configs Used to modify labels before scraping. [{"source_labels": ["__address__"], "target_label": "instance"}]

Example: Configuring Multi-Target Exporter

scrape_configs:
  - job_name: "multi_target"
    static_configs:
      - targets: ["localhost:9100", "server2:9100", "server3:9100"]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
Note: Ensure that all target exporters are running and accessible from the Prometheus server.
Warning: High cardinality from multi-target scraping can impact performance and storage.

Key Points

  • Use static_configs to define multiple targets.
  • Apply relabel_configs for label manipulation.
  • Monitor performance to avoid high cardinality issues.

Was this article helpful?

Related Cheat Sheets