Skip to main content
Mapagam

Configuring Exemplars

1. Understanding Exemplar Concept

Exemplars in Prometheus are a mechanism to associate samples with specific trace IDs or other contextual information, enhancing observability by linking metrics to distributed traces.

Property Description Values
exemplar Enables exemplar storage for metrics true / false
max_exemplars Maximum number of exemplars stored per metric family 1000

Example: Enabling Exemplars in Configuration

storage:
  exemplar:
    max_exemplars: 1000
    enabled: true
Note: Exemplars are only stored when the exemplar feature is enabled and supported by the storage backend.
Warning: Enabling exemplars can significantly increase storage requirements and query latency.

2. Enabling Exemplar Storage

Exemplars in Prometheus are used to associate samples with trace IDs, which can help in debugging and tracing issues in distributed systems. Enabling exemplar storage allows Prometheus to store these trace IDs along with the metrics.

Configuration Option Description Default Value Notes
--enable-feature=exemplar-storage Enables the exemplar storage feature in Prometheus. false This flag must be set to enable exemplar storage. It is an experimental feature.
--exemplar.max-samples-per-query Maximum number of exemplars to return per query. 100 Controls how many exemplars are returned in query results.
--exemplar.max-exemplars Maximum number of exemplars to store in memory. 10000 This setting limits the total number of exemplars stored in memory.

Example: Enabling Exemplar Storage

prometheus --enable-feature=exemplar-storage \
           --exemplar.max-samples-per-query=200 \
           --exemplar.max-exemplars=15000
Note: Exemplar storage is an experimental feature and may change in future versions of Prometheus.

3. Configuring Exemplar Retention

Configuration Option Description Default Value Notes
exemplars.max-retention Maximum duration to keep exemplars in memory before they are discarded. 3h Set to 0s to disable exemplar retention entirely.
exemplars.max-samples-per-second Maximum number of exemplars to store per second. 100 Rate-limits exemplar ingestion to prevent excessive memory usage.

Example: Prometheus Configuration with Exemplar Retention

storage:
  exemplar:
    max-retention: 2h
    max-samples-per-second: 50
Note: Exemplars are stored in memory and can consume significant resources if not configured properly. Ensure that max-samples-per-second is set to a value appropriate for your system's capacity.

4. Scraping Exemplars from Targets

Configuration Option Description Default Value Notes
scrape.exemplar Enables scraping exemplars from targets false Must be enabled to scrape exemplars
scrape.exemplar.max_samples_per_metric Maximum number of exemplars per metric 1000 Set to 0 to disable exemplar storage
scrape.exemplar.max_exemplars Maximum number of exemplars across all metrics 10000 Global limit for exemplar storage

Example: Enabling exemplar scraping in Prometheus config

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 15s
    scrape_exemplars: true
    static_configs:
      - targets: [ "localhost:9090" ]
Note: Exemplars are only scraped if the target supports them and scrape_exemplars is enabled in the job configuration.
Warning: Enabling exemplar scraping can significantly increase memory usage and storage requirements.

5. Querying Exemplars via API

Endpoint Method Description Notes
/api/v1/query_exemplars POST Query exemplars for a given instant query Requires Prometheus 2.37+

Example: Querying exemplars via API

curl -X POST "http://localhost:9090/api/v1/query_exemplars" \
  -H "Content-Type: application/json" \
  -d '{"query":"rate(http_requests_total[5m])"}'
Note: The /api/v1/query_exemplars endpoint returns exemplars associated with the result of an instant query.
Warning: Exemplars are only available for metrics that have been configured with exemplar labels.

Response Format

  • exemplars: Array of exemplar data points
  • series: Metadata about the series that the exemplars belong to

6. Using Exemplars in Histograms

Exemplars in histograms provide detailed trace information for specific metric samples, helping to debug and understand the context of high-value metric events.

Property Description Values Notes
prometheus.histogram Configures histogram with exemplar support true Enables exemplar storage for histogram metrics
exemplar.max_samples_per_metric Maximum number of exemplars per metric 100 Default limit to prevent memory bloat
exemplar.max_exemplars Global limit on exemplars in memory 10000 Global cap to control memory usage

Example: Configuring Histogram with Exemplars

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: [ "localhost:9090" ]
    exemplar:
      max_samples_per_metric: 50
      max_exemplars: 5000

Example: Histogram Metric with Exemplar

# TYPE request_duration_seconds histogram
# HELP request_duration_seconds Histogram of request durations.
request_duration_seconds_bucket{le="0.05",trace_id="abc123"} 12
request_duration_seconds_bucket{le="0.1",trace_id="def456"} 23
request_duration_seconds_bucket{le="0.5",trace_id="ghi789"} 45
request_duration_seconds_count{trace_id="jkl012"} 50
request_duration_seconds_sum{trace_id="abc123"} 4.5
Note: Exemplars are only stored for histogram buckets and counters. They do not apply to gauge or summary metrics.

7. Integrating with Tracing Systems

Prometheus supports integrating with tracing systems to provide end-to-end observability by associating metrics with trace spans. This integration allows you to correlate metric data with distributed tracing data for deeper insights into system behavior.

Integration Type Description Configuration Notes
exemplar.storage Storage backend for exemplars memory, remote Defaults to memory in Prometheus v2.35+
exemplar.max_samples_per_query Maximum number of exemplars returned per query 100 Can be adjusted to control memory usage
exemplar.max_size Maximum number of exemplars stored per series 1000 Used to limit memory consumption in memory storage

Example: Enabling exemplar storage in Prometheus config

storage:
  exemplar:
    max_size: 10000
    storage: memory
Note: Exemplar storage is experimental and may change in future versions. Ensure that your tracing system supports Prometheus exemplars to fully utilize this feature.

Tracing System Integration Options

  • OpenTelemetry: Supports exemplar export via OTLP
  • Jaeger: Can receive exemplars through Prometheus remote write
  • Zipkin: Requires custom integration or remote write support

Integration Workflow

  1. Configure Prometheus to store exemplars in memory or remote storage
  2. Set up tracing system (e.g., OpenTelemetry Collector) to receive exemplars
  3. Ensure tracing system supports Prometheus exemplar format
  4. Query metrics with exemplars to correlate with trace data

8. Configuring Exemplar Limits

Option Values Notes
exemplar.max_samples_per_query 1000 Maximum number of exemplars returned per query
exemplar.max_exemplars 100000 Maximum number of exemplars stored in memory

Example: Setting exemplar limits in Prometheus configuration

scrape_configs:
  - job_name: 'prometheus'
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    scheme: http
    static_configs:
      - targets: ['localhost:9090']
    exemplar:
      max_samples_per_query: 1000
      max_exemplars: 100000
Note: Exemplars are stored in memory and can consume significant resources if not properly limited.

9. Visualizing Exemplars

Example: Grafana Dashboard with Exemplars

{
  "__inputs": [
    {
      "name": "DS_PROMETHEUS",
      "label": "Prometheus",
      "description": "",
      "pluginId": "prometheus",
      "pluginName": "Prometheus"
    }
  ],
  "__requires": [
    {
      "type": "panel",
      "id": "graph",
      "name": "Graph (old)",
      "version": ""
    },
    {
      "type": "datasource",
      "id": "prometheus",
      "name": "Prometheus",
      "version": "1.0.0"
    }
  ],
  "annotations": {
    "list": []
  },
  "editable": true,
  "gnetId": null,
  "graphTooltip": 1,
  "id": null,
  "links": [],
  "panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": "${DS_PROMETHEUS}",
      "fill": 1,
      "gridPos": {
        "h": 9,
        "w": 12,
        "x": 0,
        "y": 0
      },
      "id": 1,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "links": [],
      "nullPointMode": "null",
      "percentage": false,
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "span": 1,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "expr": "rate(http_requests_total[5m])",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{method}} {{path}}",
          "refId": "A"
        }
      ],
      "thresholds": [],
      "timeFrom": null,
      "timeShift": null,
      "title": "HTTP Request Rate with Exemplars",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "name": null,
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "format": "ops",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": 0,
          "show": true
        },
        {
          "format": "short",
          "label": null,
          "logBase": 1,
          "max": null,
          "min": null,
          "show": true
        }
      ],
      "yaxis": {
        "align": false,
        "alignLevel": null
      }
    }
  ],
  "schemaVersion": 16,
  "style": "dark",
  "tags": [
    "prometheus"
  ],
  "templating": {
    "list": []
  },
  "time": {
    "from": "now-1h",
    "to": "now"
  },
  "timepicker": {
    "refresh_intervals": [
      "5s",
      "10s",
      "30s",
      "1m",
      "5m",
      "15m",
      "30m",
      "1h",
      "2h",
      "1d"
    ],
    "time_options": [
      "5m",
      "15m",
      "1h",
      "6h",
      "12h",
      "24h",
      "2d",
      "7d",
      "30d"
    ]
  },
  "timezone": "",
  "title": "Prometheus Exemplars Dashboard",
  "version": 1
}
Feature Description Notes
Exemplar Support Support for storing and retrieving exemplars in Prometheus Requires Prometheus 2.32+ for full support
Grafana Integration Visualizing exemplars in Grafana dashboards Exemplars appear as small dots on graphs
Querying Exemplars Using PromQL to query exemplars Use exemplar function in queries
Note: Exemplars are only supported in Prometheus 2.32 and later versions.
Warning: Enabling exemplars can increase storage requirements significantly.

Key Points for Visualization

  • Exemplars are visualized as small dots on graphs in Grafana
  • They provide context to metric values by showing actual sample data
  • Only Prometheus 2.32+ supports exemplars

10. Monitoring Exemplar Storage

Property Description Default Value Notes
exemplar.storage Storage backend for exemplars memory Valid values: memory, remote
exemplar.max_samples_per_query Maximum number of exemplars to return per query 100 Controls memory usage during query processing
exemplar.max_exemplars Maximum number of exemplars to store in memory 10000 Applies only to memory storage

Example: Configuring exemplar storage in Prometheus

storage:
  exemplar:
    storage: memory
    max_samples_per_query: 50
    max_exemplars: 5000
Note: Exemplars are only stored in memory by default. For production use, consider remote storage or a dedicated exemplar backend.

11. Implementing Exemplar Best Practices

Example: Configuring Exemplar Storage

storage:
  exemplars:
    max_exemplars: 100000
    retention_duration: 1h
Configuration Description Default Value Notes
max_exemplars Maximum number of exemplars to store 100000 Set to 0 to disable exemplar storage
retention_duration How long to retain exemplars 1h Duration in seconds or with unit suffixes (e.g., 1h, 30m)

Best Practices for Exemplar Configuration

  • Set max_exemplars based on available memory and desired cardinality
  • Use retention_duration to balance storage cost and query utility
  • Monitor exemplar storage usage via prometheus_storage_exemplars_in_memory metric
  • Enable exemplar storage only for metrics that benefit from detailed tracing
Warning: Enabling exemplar storage can significantly increase memory usage and storage requirements. Ensure sufficient resources before enabling.

12. Using Exemplars for Debugging

Property Description Example
exemplar Enables exemplar storage for metrics true
max_exemplars Maximum number of exemplars to store per metric family 1000

Example: Enabling Exemplar Storage

storage:
  exemplar:
    max_exemplars: 1000
Note: Exemplars are stored in-memory and can consume significant memory if not managed properly.

Key Points

  • Exemplars provide trace IDs for metrics to help with debugging.
  • They are particularly useful in distributed systems where tracing is enabled.
  • Configuring max_exemplars helps manage memory usage.

Was this article helpful?

Related Cheat Sheets