Skip to main content

Implementing Native Histograms

1. Understanding Native Histograms

Native histograms are a feature introduced in Prometheus 2.20, designed to provide more efficient storage and querying of histogram data compared to traditional histograms.

Feature Description Notes
Storage Efficiency Native histograms store data in a compressed format using a binary tree structure. Reduces storage space compared to classic histograms.
Query Performance Supports efficient aggregation and quantile calculations. Enhanced performance for queries involving histogram data.
Bucket Handling Buckets are not explicitly defined; instead, they are dynamically created based on data distribution. Eliminates the need for predefined bucket boundaries.
Data Granularity Supports high granularity with up to 128 buckets per histogram. Allows for more precise metric analysis.

Example: Histogram Configuration

native_histograms:
  max_buckets: 128
  min_age: 1h
Note: Native histograms are still experimental and may change in future versions.
Warning: When using native histograms, ensure that your Prometheus version supports them (2.20+).

2. Enabling Native Histograms

Native histograms are a feature introduced in Prometheus 2.32.0 to improve the efficiency of histogram metrics.

Configuration Option Description Default Value Notes
--enable-feature=native-histograms Enables the native histogram feature false Flag must be set to enable native histograms

Example: Enabling Native Histograms via CLI

prometheus --enable-feature=native-histograms
Note: Native histograms are still experimental and may change in future versions.
Warning: Enabling native histograms may cause compatibility issues with older versions of Prometheus or related tools.

3. Scraping Native Histograms

Configuration Option Description Default Value Notes
native_histograms Enables scraping native histograms false Must be enabled to scrape native histograms
histogram_buckets Defines bucket boundaries for native histograms auto-generated Can be customized per metric family

Example: Configuring Native Histograms in Prometheus

scrape_configs:
  - job_name: "prometheus"
    scrape_interval: 15s
    metrics_path: /metrics
    native_histograms: true
    static_configs:
      - targets: [localhost:9090]
Note: Native histograms are supported in Prometheus v2.37+ and require the native_histograms feature flag to be enabled.
Warning: When scraping native histograms, ensure that the Prometheus server has sufficient memory to handle the histogram data.

4. Querying Native Histograms

Native histograms in Prometheus can be queried using the same PromQL syntax as regular histograms, with some additional functions and considerations specific to their structure.

Function Description Example
histogram_quantile() Calculates quantiles from histogram data, including native histograms. histogram_quantile(0.9, native_histogram_metric)
histogram_fraction() Calculates the fraction of observations within a range, useful for native histograms. histogram_fraction(0.5, 1.0, native_histogram_metric)
histogram_count() Returns the count of observations in a histogram, including native histograms. histogram_count(native_histogram_metric)
histogram_sum() Returns the sum of observations in a histogram, including native histograms. histogram_sum(native_histogram_metric)

Example: Calculating Quantiles from Native Histograms

histogram_quantile(0.95, http_request_duration_seconds)

This query calculates the 95th percentile of HTTP request durations, using native histogram data.

Example: Comparing Histograms with Different Buckets

histogram_quantile(0.9, native_histogram_metric) / histogram_quantile(0.9, regular_histogram_metric)

Compares quantiles between native and traditional histogram metrics to assess differences in distribution.

5. Using Histogram Operators

Operator Description Example Notes
histogram_quantile() Calculates quantiles from histogram data histogram_quantile(0.95, my_histogram) Useful for latency SLOs
histogram_fraction() Calculates fraction of histogram values within a range histogram_fraction(0.5, 1.0, my_histogram) Can be used for percentiles
histogram_count() Returns the count of observations in a histogram histogram_count(my_histogram) Useful for monitoring total requests
histogram_sum() Returns the sum of observations in a histogram histogram_sum(my_histogram) Useful for calculating average

Example: Calculating 95th Percentile

histogram_quantile(0.95, http_request_duration_seconds)

Example: Histogram with Custom Buckets

histogram_quantile(0.99, http_request_duration_seconds_bucket)

6. Configuring Bucket Schema

Option Values Description
schema 0, 1, ..., 12 Defines the bucket schema for native histograms. Higher values provide more granularity but increase memory usage.
max_schema 0 to 12 Maximum schema value allowed for native histograms. Default is 12.
min_schema 0 to 12 Minimum schema value allowed for native histograms. Default is 0.

Example: Setting bucket schema in Prometheus configuration

prometheus:
  enable_native_histograms: true
  histogram_config:
    schema: 1
Note: Schema 0 is the default and provides basic bucket configuration. Higher schemas (e.g., 1 to 12) offer more granular bucket definitions for better histogram accuracy.
Warning: Setting a high schema value (e.g., 12) can significantly increase memory usage and may impact performance.

7. Converting Classic to Native

Option Description Notes
native_histogram_bucket_limit 60 Maximum number of buckets in a native histogram. Default is 60.
native_histogram_min_bucket_width 1 Minimum width of a bucket in a native histogram. Default is 1.
native_histogram_max_bucket_width 1000000 Maximum width of a bucket in a native histogram. Default is 1,000,000.

Example: Converting Classic Histogram to Native

groups:
- name: example
  rules:
  - record: http_requests_native
    expr: histogram_quantile(0.95, sum by (le) (rate(http_requests_total[5m])))
Note: Native histograms are a new feature in Prometheus v2.35+ and provide better performance for high-cardinality metrics.

8. Calculating Aggregations

Aggregation Type Description Formula Notes
Sum Total of all values in the histogram sum = Σ(values) Used for total metric calculation
Count Number of observations in the histogram count = Σ(observations) Used to determine sample size
Average Average value of all observations avg = sum / count Calculated from sum and count
Quantiles Percentile values (e.g., 50th, 90th, 99th) quantile = interpolation of histogram bins Requires bin data to compute

Example: Calculating Sum and Count

const histogram = {
  buckets: [10, 20, 30, 40, 50],
  counts: [5, 10, 15, 8, 2]
};

const sum = histogram.buckets.reduce((acc, bucket, i) => acc + (bucket * histogram.counts[i]), 0);
const count = histogram.counts.reduce((acc, c) => acc + c, 0);

console.log("Sum:", sum); // Sum: 10*5 + 20*10 + 30*15 + 40*8 + 50*2 = 50 + 200 + 450 + 320 + 100 = 1120
console.log("Count:", count); // Count: 40

Example: Calculating Quantiles

function calculateQuantile(histogram, q) {
  const total = histogram.counts.reduce((a, b) => a + b, 0);
  const target = q * total;
  let cumulative = 0;
  for (let i = 0; i < histogram.buckets.length; i++) {
    cumulative += histogram.counts[i];
    if (cumulative >= target) {
      return histogram.buckets[i];
    }
  }
  return histogram.buckets[histogram.buckets.length - 1];
}

const q90 = calculateQuantile(histogram, 0.9);
console.log("90th Percentile:", q90); // 40

9. Implementing Histogram Merging

Histogram merging is a critical operation in distributed Prometheus setups where histogram data needs to be aggregated across multiple instances. This process ensures that the final aggregated histogram accurately represents the combined distribution of data from all sources.

Operation Description Notes
MergeHistograms Merges multiple histogram samples into a single histogram Must maintain bucket boundaries and counts correctly
Bucket Alignment Ensures bucket boundaries are aligned across instances Required for accurate merging; mismatched buckets cause data loss
Count and Sum Aggregation Aggregates total count and sum of values Sum is aggregated directly, count is summed across buckets

Example: Merging Histograms in Go

import "github.com/prometheus/client_model/go"

func MergeHistograms(hists []*client_model.Histogram) *client_model.Histogram {
    if len(hists) == 0 {
        return nil
    }

    // Initialize result with first histogram's schema and buckets
    result := &client_model.Histogram{
        Schema:     hists[0].Schema,
        ZeroThreshold: hists[0].ZeroThreshold,
        ZeroCount:  hists[0].ZeroCount,
        Count:      hists[0].Count,
        Sum:        hists[0].Sum,
    }

    // Merge buckets and aggregate counts
    for i := 1; i < len(hists); i++ {
        // Merge bucket counts and update sum
        result.Count += hists[i].Count
        result.Sum += hists[i].Sum
    }

    return result
}
Note: When merging histograms, it's crucial to ensure that all input histograms use the same bucket schema. Mismatched schemas can lead to incorrect aggregation and loss of data fidelity.
Warning: Histograms with different bucket boundaries cannot be merged directly. They must first be aligned using a common bucket schema.

10. Monitoring Native Histogram Storage

Property Description Values Notes
prometheus_native_histogram_bucket_limit Maximum number of buckets in a native histogram 256 Default limit; can be increased up to 1024
prometheus_native_histogram_max_age Maximum age of a native histogram bucket 24h Defaults to 24 hours; affects data retention
prometheus_native_histogram_min_bucket Minimum bucket size in a native histogram 1e-09 Smallest bucket size for native histogram

Example: Configuring Native Histogram Storage

prometheus:
  storage:
    native_histograms:
      bucket_limit: 512
      max_age: 48h
      min_bucket: 1e-08
Note: Native histograms are a new feature in Prometheus v2.35+ and require enabling via the --enable-feature=native-histograms flag.
Warning: Increasing bucket_limit can significantly increase memory usage.

11. Understanding Bucket Resolution

Resolution Level Description Bucket Count Notes
0 Coarsest resolution, 1 bucket per order of magnitude 20 buckets Default for native histograms
1 Medium resolution, 2 buckets per order of magnitude 40 buckets Good balance for most use cases
2 High resolution, 5 buckets per order of magnitude 100 buckets Best for detailed analysis
3 Ultra-high resolution, 10 buckets per order of magnitude 200 buckets For very high precision needs

Example: Setting Bucket Resolution in Prometheus

native_histogram_bucket_limit: 200
native_histogram_max_bucket: 1000000
Note: Higher resolution increases memory usage and query time. Choose based on your precision needs.

12. Implementing Custom Bucket Boundaries

Configuration Option Description Values Notes
bucket Defines custom bucket boundaries for histogram [10, 25, 50, 100] Must be in ascending order
native_histogram_bucket Defines bucket boundaries for native histogram [1, 2, 4, 8] Used in native histogram implementation

Example: Defining Custom Buckets in Prometheus

groups:
- name: example
  rules:
  - alert: HighLatency
    expr: histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m])) > 1
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "High latency detected"
      description: "The 95th percentile request duration is above 1 second."
Note: Custom bucket boundaries should be chosen based on the expected distribution of your metrics to ensure accurate quantile calculations.

Was this article helpful?