Working with OpenMetrics Format

1. Understanding OpenMetrics Specification

Component	Description	Examples
Metric Name	Identifier for a metric, must follow specific naming rules	`http_requests_total`
Metric Type	Defines the kind of data being collected (gauge, counter, histogram, etc.)	`counter`, `gauge`, `histogram`
Labels	Key-value pairs to provide additional context to a metric	`{method="GET", status="200"}`
Help Text	Human-readable description of the metric	`# HELP http_requests_total Total number of HTTP requests`
Unit	Optional unit for the metric (e.g., seconds, bytes)	`# UNIT http_requests_total requests`

Example: Basic Metric Format

# HELP http_requests_total Total number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET", status="200"} 1234

Note: OpenMetrics is a superset of the Prometheus exposition format and includes additional features like unit support and better handling of timestamps.

2. Scraping OpenMetrics Endpoints

Configuration Option	Description	Default Value	Notes
scrape_configs	Defines how to scrape targets	None	Required field in Prometheus config
job_name	Unique identifier for the job	None	Must be unique across all jobs
metrics_path	Path to scrape metrics from	`/metrics`	Can be overridden in config
scrape_interval	How often to scrape targets	`15s`	Can be overridden per job
scrape_timeout	Timeout for scraping targets	`10s`	Must be less than scrape_interval
honor_labels	Whether to honor labels from scraped data	`false`	Set to `true` to prevent label conflicts
relabel_configs	Rules to modify labels before scraping	None	Used for filtering or renaming labels

Example: Basic Scrape Configuration

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: [localhost:9090]

Example: Scrape with Relabeling

scrape_configs:
  - job_name: "app"
    static_configs:
      - targets: [localhost:8080]
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance

Note: Prometheus supports both text/plain and application/openmetrics-text content types for scraping endpoints.

Warning: Ensure that the scrape_timeout is less than scrape_interval to avoid overlapping scrapes.

3. Using Typed Metrics

Metric Type	Description	Example	Notes
counter	Counts increases over time, typically for events like requests or errors.	`# HELP http_requests_total Total number of HTTP requests # TYPE http_requests_total counter http_requests_total{method="GET",code="200"} 1234`	Must be non-negative and monotonically increasing.
gauge	Represents a single numerical value that can go up and down.	`# HELP memory_usage_bytes Current memory usage in bytes # TYPE memory_usage_bytes gauge memory_usage_bytes 1048576`	Can be set to any value, including negative.
histogram	Tracks the distribution of values, typically used for latency or size measurements.	`# HELP request_duration_seconds Histogram of request durations # TYPE request_duration_seconds histogram request_duration_seconds_bucket{le="0.05"} 100 request_duration_seconds_bucket{le="0.1"} 250 request_duration_seconds_sum 45.3 request_duration_seconds_count 500`	Includes bucketed counts and sum for percentile calculations.
summary	Similar to histogram but provides quantiles (e.g., 90th percentile) directly.	`# HELP response_time_seconds Summary of response times # TYPE response_time_seconds summary response_time_seconds{quantile="0.5"} 0.025 response_time_seconds{quantile="0.9"} 0.052 response_time_seconds_sum 123.456 response_time_seconds_count 1000`	Quantiles are defined by the client or server.
untyped	Used for metrics without a defined type, often for compatibility or unknown types.	`# HELP custom_metric Untyped metric example # TYPE custom_metric untyped custom_metric 42.0`	Not recommended for new metrics; prefer explicit types.

Example: Counter Usage in a Web Server

# HELP http_requests_total Number of HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="POST",code="200"} 15
http_requests_total{method="GET",code="200"} 120
http_requests_total{method="GET",code="404"} 5

Example: Histogram with Buckets

# HELP latency_seconds Histogram of latency in seconds
# TYPE latency_seconds histogram
latency_seconds_bucket{le="0.01"} 50
latency_seconds_bucket{le="0.05"} 120
latency_seconds_bucket{le="0.1"} 200
latency_seconds_bucket{le="+Inf"} 300
latency_seconds_sum 25.6
latency_seconds_count 300

4. Understanding Metric Units

Metric Unit	Description	Examples	Notes
bytes	Represents binary byte measurements	`1024`	Used for memory, disk space, bandwidth
seconds	Time in seconds, typically for durations or latencies	`0.5`	Common for response times, uptime
count	Represents a count of things, usually without unit suffix	`100`	Used for counters, gauge values
percent	Percentage values, typically 0-100 or 0.0-1.0	`95.5`	Often used for CPU usage, success rates
bytes/second	Data rate in bytes per second	`102400`	Useful for bandwidth monitoring
requests/second	Request rate per second	`50`	Used for API or service load metrics

Example: Using Units in Prometheus Metrics


memory_usage_bytes 1048576
http_request_duration_seconds 0.25
error_count 10
cpu_utilization_percent 75.3
network_throughput_bytes_per_second 204800

Note: Prometheus does not enforce unit types but encourages consistent usage for clarity and interoperability with tools that understand metric semantics.

5. Working with Exemplars

Exemplars in Prometheus are additional data points that provide context to histogram and summary metrics, helping to identify the source of high latency or errors.

Field	Description	Example
exemplar	Represents a sample with additional context, typically used in histograms and summaries.	`{trace_id="1234567890abcdef", span_id="0987654321fedcba"}`

Example: Adding Exemplars to Histograms

# HELP request_duration_seconds Histogram of request durations
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.05", trace_id="1234567890abcdef"} 10
request_duration_seconds_bucket{le="0.1", trace_id="1234567890abcdef"} 15
request_duration_seconds_bucket{le="0.2", trace_id="1234567890abcdef"} 20
request_duration_seconds_bucket{le="+Inf", trace_id="1234567890abcdef"} 25
request_duration_seconds_sum{trace_id="1234567890abcdef"} 2.5
request_duration_seconds_count{trace_id="1234567890abcdef"} 25

Note: Exemplars are only supported in Prometheus 2.32+ and must be enabled via the --enable-feature=exemplar-storage flag.

Warning: Exemplars can significantly increase the size of your metrics data. Ensure that you have sufficient storage and query performance considerations.

6. Using Info Metrics

Field	Description	Example
metric_name	The name of the metric, followed by `{}` for labels	`node_info{instance="localhost:9100",job="node"}`
labels	Key-value pairs providing metadata about the metric	`instance="localhost:9100",job="node"`
value	Always `1` for info metrics (they represent static information)	`1`

Example: Info Metric Definition

# HELP node_info Node information
# TYPE node_info info
node_info{instance="localhost:9100",job="node"} 1

Note: Info metrics are used to expose static information like version numbers, build timestamps, or system details. They do not change over time.

Warning: Do not use info metrics for dynamic data. Prometheus will not handle changes in labels or values correctly.

7. Understanding StateSet Metrics

Metric Type	Description	Usage	Example
state	Represents a boolean value indicating the state of a system or component.	Used to track whether a condition is true or false, such as if a service is up or down.	`service_state{service="auth",state="up"} 1 service_state{service="auth",state="down"} 0`
status	Indicates the operational status of a system or component, often used in conjunction with state.	Used to represent the status of a system (e.g., healthy, degraded, critical).	`system_status{system="database",status="healthy"} 1 system_status{system="database",status="degraded"} 0`

Example: StateSet Metrics in Prometheus

# Example of a Prometheus recording rule that aggregates state metrics
groups:
- name: service_states
  rules:
  - record: service_state:up
    expr: up{job="service"} == 1

Note: StateSet metrics are typically used in alerting rules to monitor system health.

Key Points About StateSet Metrics

StateSet metrics are a type of gauge metric that represents a boolean state.
They are useful for tracking the operational status of services or components.
StateSet metrics can be used in combination with other metrics to provide a more comprehensive view of system health.

8. Using Histogram Buckets

Histograms in Prometheus are used to track the distribution of values, typically for latency or size measurements. Buckets define the upper bounds of each bucket in a histogram.

Bucket Bound	Description	Example
le	Upper bound of the bucket (less than or equal to)	`le="0.05"`
le="+Inf"	Represents the infinite upper bound, capturing all values	`le="+Inf"`

Example: Histogram with Custom Buckets

# HELP request_duration_seconds Histogram of request durations
# TYPE request_duration_seconds histogram
request_duration_seconds_bucket{le="0.05"} 10
request_duration_seconds_bucket{le="0.1"} 25
request_duration_seconds_bucket{le="0.5"} 40
request_duration_seconds_bucket{le="1.0"} 50
request_duration_seconds_bucket{le="+Inf"} 60
request_duration_seconds_sum 35.0
request_duration_seconds_count 60

Note: The le label represents the upper bound of each bucket, and the +Inf bucket captures all values.

Example: Histogram with Default Buckets

# HELP http_request_duration_seconds Histogram of HTTP request durations
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.005"} 100
http_request_duration_seconds_bucket{le="0.01"} 250
http_request_duration_seconds_bucket{le="0.025"} 400
http_request_duration_seconds_bucket{le="0.05"} 500
http_request_duration_seconds_bucket{le="0.1"} 600
http_request_duration_seconds_bucket{le="0.25"} 700
http_request_duration_seconds_bucket{le="0.5"} 800
http_request_duration_seconds_bucket{le="1.0"} 900
http_request_duration_seconds_bucket{le="2.5"} 1000
http_request_duration_seconds_bucket{le="5.0"} 1100
http_request_duration_seconds_bucket{le="10.0"} 1200
http_request_duration_seconds_bucket{le="+Inf"} 1250
http_request_duration_seconds_sum 6500.0
http_request_duration_seconds_count 1250

9. Understanding EOF Marker

Note: The EOF marker is not part of the OpenMetrics format specification but is a convention used by some tools to indicate the end of a metrics stream.

Marker	Description	Usage
eof	Indicates the end of the metrics stream	Used by some exporters and scrapers to signal completion

Example: EOF Marker in Metrics Stream

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="get",code="200"} 100
http_requests_total{method="post",code="201"} 50
# EOF

10. Configuring Accept Header

Header	Value	Description
Accept	`application/openmetrics-text; version=1.0.0; charset=utf-8`	Specifies the OpenMetrics format with version and encoding
Accept	`application/openmetrics-text; version=0.0.1; charset=utf-8`	Alternative OpenMetrics version 0.0.1
Accept	`application/openmetrics-text; version=1.0.0`	OpenMetrics format without explicit charset
Accept	`application/openmetrics-text; version=0.0.1`	OpenMetrics 0.0.1 format without charset

Example: Setting Accept Header in cURL

curl -H "Accept: application/openmetrics-text; version=1.0.0; charset=utf-8" http://prometheus-server/metrics

11. Implementing Created Timestamps

Created timestamps in OpenMetrics format are used to indicate when a metric was first created or initialized. They help in tracking the lifetime of metrics and are particularly useful for long-running systems.

Field	Description	Example
metric_name_created	Indicates the creation time of a metric in Unix timestamp format (seconds since epoch)	`1609459200`

Example: Adding Created Timestamp to a Metric

# TYPE http_requests_total counter
# HELP http_requests_total Total number of HTTP requests
http_requests_total 12345
http_requests_total_created 1609459200

Note: The _created suffix is used to denote the creation timestamp for a metric. This is part of the OpenMetrics specification and ensures compatibility with tools that support this format.

12. Understanding OpenMetrics vs Prometheus Format

Feature	OpenMetrics	Prometheus Format
Specification	Standardized by OpenMetrics initiative, version 1.0.0	Proprietary to Prometheus, not standardized
MIME Type	`application/openmetrics-text; version=1.0.0`	`text/plain; version=0.0.4`
Data Types Supported	Counter, Gauge, Histogram, Summary, Info, StateSet	Counter, Gauge, Histogram, Summary, Untyped
Label Handling	Enforces strict label escaping and validation	Allows less strict label handling
Metric Metadata	Supports metric metadata with `# TYPE` and `# HELP`	Supports metric metadata with `# TYPE` and `# HELP`
Timestamps	Supports optional timestamps with `@` syntax	Does not support timestamps
Extensions	Supports extensions via `# EOF` and custom headers	Does not support extensions

Note: OpenMetrics is a superset of the Prometheus exposition format and aims to provide a standardized way to expose metrics.