Implementing Native Histograms
1. Understanding Native Histograms
Native histograms are a feature introduced in Prometheus 2.20, designed to provide more efficient storage and querying of histogram data compared to traditional histograms.
| Feature | Description | Notes |
|---|---|---|
| Storage Efficiency | Native histograms store data in a compressed format using a binary tree structure. | Reduces storage space compared to classic histograms. |
| Query Performance | Supports efficient aggregation and quantile calculations. | Enhanced performance for queries involving histogram data. |
| Bucket Handling | Buckets are not explicitly defined; instead, they are dynamically created based on data distribution. | Eliminates the need for predefined bucket boundaries. |
| Data Granularity | Supports high granularity with up to 128 buckets per histogram. | Allows for more precise metric analysis. |
2. Enabling Native Histograms
Native histograms are a feature introduced in Prometheus 2.32.0 to improve the efficiency of histogram metrics.
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| --enable-feature=native-histograms | Enables the native histogram feature | false | Flag must be set to enable native histograms |
3. Scraping Native Histograms
| Configuration Option | Description | Default Value | Notes |
|---|---|---|---|
| native_histograms | Enables scraping native histograms | false | Must be enabled to scrape native histograms |
| histogram_buckets | Defines bucket boundaries for native histograms | auto-generated | Can be customized per metric family |
Example: Configuring Native Histograms in Prometheus
scrape_configs:
- job_name: "prometheus"
scrape_interval: 15s
metrics_path: /metrics
native_histograms: true
static_configs:
- targets: [localhost:9090]
native_histograms feature flag to be enabled.
4. Querying Native Histograms
Native histograms in Prometheus can be queried using the same PromQL syntax as regular histograms, with some additional functions and considerations specific to their structure.
| Function | Description | Example |
|---|---|---|
| histogram_quantile() | Calculates quantiles from histogram data, including native histograms. | histogram_quantile(0.9, native_histogram_metric) |
| histogram_fraction() | Calculates the fraction of observations within a range, useful for native histograms. | histogram_fraction(0.5, 1.0, native_histogram_metric) |
| histogram_count() | Returns the count of observations in a histogram, including native histograms. | histogram_count(native_histogram_metric) |
| histogram_sum() | Returns the sum of observations in a histogram, including native histograms. | histogram_sum(native_histogram_metric) |
Example: Calculating Quantiles from Native Histograms
histogram_quantile(0.95, http_request_duration_seconds)
This query calculates the 95th percentile of HTTP request durations, using native histogram data.
Example: Comparing Histograms with Different Buckets
histogram_quantile(0.9, native_histogram_metric) / histogram_quantile(0.9, regular_histogram_metric)
Compares quantiles between native and traditional histogram metrics to assess differences in distribution.
5. Using Histogram Operators
| Operator | Description | Example | Notes |
|---|---|---|---|
| histogram_quantile() | Calculates quantiles from histogram data | histogram_quantile(0.95, my_histogram) |
Useful for latency SLOs |
| histogram_fraction() | Calculates fraction of histogram values within a range | histogram_fraction(0.5, 1.0, my_histogram) |
Can be used for percentiles |
| histogram_count() | Returns the count of observations in a histogram | histogram_count(my_histogram) |
Useful for monitoring total requests |
| histogram_sum() | Returns the sum of observations in a histogram | histogram_sum(my_histogram) |
Useful for calculating average |
Example: Histogram with Custom Buckets
histogram_quantile(0.99, http_request_duration_seconds_bucket)
6. Configuring Bucket Schema
| Option | Values | Description |
|---|---|---|
| schema | 0, 1, ..., 12 |
Defines the bucket schema for native histograms. Higher values provide more granularity but increase memory usage. |
| max_schema | 0 to 12 |
Maximum schema value allowed for native histograms. Default is 12. |
| min_schema | 0 to 12 |
Minimum schema value allowed for native histograms. Default is 0. |
Example: Setting bucket schema in Prometheus configuration
prometheus:
enable_native_histograms: true
histogram_config:
schema: 1
0 is the default and provides basic bucket configuration. Higher schemas (e.g., 1 to 12) offer more granular bucket definitions for better histogram accuracy.
12) can significantly increase memory usage and may impact performance.
7. Converting Classic to Native
| Option | Description | Notes |
|---|---|---|
| native_histogram_bucket_limit | 60 |
Maximum number of buckets in a native histogram. Default is 60. |
| native_histogram_min_bucket_width | 1 |
Minimum width of a bucket in a native histogram. Default is 1. |
| native_histogram_max_bucket_width | 1000000 |
Maximum width of a bucket in a native histogram. Default is 1,000,000. |
Example: Converting Classic Histogram to Native
groups:
- name: example
rules:
- record: http_requests_native
expr: histogram_quantile(0.95, sum by (le) (rate(http_requests_total[5m])))
8. Calculating Aggregations
| Aggregation Type | Description | Formula | Notes |
|---|---|---|---|
| Sum | Total of all values in the histogram | sum = Σ(values) |
Used for total metric calculation |
| Count | Number of observations in the histogram | count = Σ(observations) |
Used to determine sample size |
| Average | Average value of all observations | avg = sum / count |
Calculated from sum and count |
| Quantiles | Percentile values (e.g., 50th, 90th, 99th) | quantile = interpolation of histogram bins |
Requires bin data to compute |
Example: Calculating Sum and Count
const histogram = {
buckets: [10, 20, 30, 40, 50],
counts: [5, 10, 15, 8, 2]
};
const sum = histogram.buckets.reduce((acc, bucket, i) => acc + (bucket * histogram.counts[i]), 0);
const count = histogram.counts.reduce((acc, c) => acc + c, 0);
console.log("Sum:", sum); // Sum: 10*5 + 20*10 + 30*15 + 40*8 + 50*2 = 50 + 200 + 450 + 320 + 100 = 1120
console.log("Count:", count); // Count: 40
Example: Calculating Quantiles
function calculateQuantile(histogram, q) {
const total = histogram.counts.reduce((a, b) => a + b, 0);
const target = q * total;
let cumulative = 0;
for (let i = 0; i < histogram.buckets.length; i++) {
cumulative += histogram.counts[i];
if (cumulative >= target) {
return histogram.buckets[i];
}
}
return histogram.buckets[histogram.buckets.length - 1];
}
const q90 = calculateQuantile(histogram, 0.9);
console.log("90th Percentile:", q90); // 40
9. Implementing Histogram Merging
Histogram merging is a critical operation in distributed Prometheus setups where histogram data needs to be aggregated across multiple instances. This process ensures that the final aggregated histogram accurately represents the combined distribution of data from all sources.
| Operation | Description | Notes |
|---|---|---|
| MergeHistograms | Merges multiple histogram samples into a single histogram | Must maintain bucket boundaries and counts correctly |
| Bucket Alignment | Ensures bucket boundaries are aligned across instances | Required for accurate merging; mismatched buckets cause data loss |
| Count and Sum Aggregation | Aggregates total count and sum of values | Sum is aggregated directly, count is summed across buckets |
Example: Merging Histograms in Go
import "github.com/prometheus/client_model/go"
func MergeHistograms(hists []*client_model.Histogram) *client_model.Histogram {
if len(hists) == 0 {
return nil
}
// Initialize result with first histogram's schema and buckets
result := &client_model.Histogram{
Schema: hists[0].Schema,
ZeroThreshold: hists[0].ZeroThreshold,
ZeroCount: hists[0].ZeroCount,
Count: hists[0].Count,
Sum: hists[0].Sum,
}
// Merge buckets and aggregate counts
for i := 1; i < len(hists); i++ {
// Merge bucket counts and update sum
result.Count += hists[i].Count
result.Sum += hists[i].Sum
}
return result
}
10. Monitoring Native Histogram Storage
| Property | Description | Values | Notes |
|---|---|---|---|
| prometheus_native_histogram_bucket_limit | Maximum number of buckets in a native histogram | 256 |
Default limit; can be increased up to 1024 |
| prometheus_native_histogram_max_age | Maximum age of a native histogram bucket | 24h |
Defaults to 24 hours; affects data retention |
| prometheus_native_histogram_min_bucket | Minimum bucket size in a native histogram | 1e-09 |
Smallest bucket size for native histogram |
Example: Configuring Native Histogram Storage
prometheus:
storage:
native_histograms:
bucket_limit: 512
max_age: 48h
min_bucket: 1e-08
--enable-feature=native-histograms flag.
bucket_limit can significantly increase memory usage.
11. Understanding Bucket Resolution
| Resolution Level | Description | Bucket Count | Notes |
|---|---|---|---|
| 0 | Coarsest resolution, 1 bucket per order of magnitude | 20 buckets | Default for native histograms |
| 1 | Medium resolution, 2 buckets per order of magnitude | 40 buckets | Good balance for most use cases |
| 2 | High resolution, 5 buckets per order of magnitude | 100 buckets | Best for detailed analysis |
| 3 | Ultra-high resolution, 10 buckets per order of magnitude | 200 buckets | For very high precision needs |
Example: Setting Bucket Resolution in Prometheus
native_histogram_bucket_limit: 200
native_histogram_max_bucket: 1000000
12. Implementing Custom Bucket Boundaries
| Configuration Option | Description | Values | Notes |
|---|---|---|---|
| bucket | Defines custom bucket boundaries for histogram | [10, 25, 50, 100] |
Must be in ascending order |
| native_histogram_bucket | Defines bucket boundaries for native histogram | [1, 2, 4, 8] |
Used in native histogram implementation |
Example: Defining Custom Buckets in Prometheus
groups:
- name: example
rules:
- alert: HighLatency
expr: histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m])) > 1
for: 10m
labels:
severity: page
annotations:
summary: "High latency detected"
description: "The 95th percentile request duration is above 1 second."