Skip to main content
Mapagam

Working with Kubernetes Integration

1. Using Kubernetes Service Discovery

Property Description Example
kubernetes_sd_configs Configuration for Kubernetes service discovery 
[{
  "role": "pod"
}]
role Role to discover (pod, service, endpoints, node) "pod"
namespaces Namespaces to watch for services ["default"]

Example: Service Discovery Configuration

scrape_configs:
- job_name: "kubernetes-pods"
  kubernetes_sd_configs:
  - role: pod
    namespaces:
      names:
      - default
Note: Prometheus uses the Kubernetes API server to discover services and pods. Ensure proper RBAC permissions are configured for the Prometheus service account.

2. Discovering Pods

Property Description Example
pod.name Name of the pod my-app-pod-12345
pod.namespace Namespace where the pod resides default
pod.status Current status of the pod (Running, Pending, Failed) Running
pod.ip IP address of the pod 10.244.0.5
pod.node Node name where the pod is scheduled kube-node-01

Example: Querying Pod Metrics

prometheus_remote_write {
  url = "http://prometheus:9090/api/v1/write"
  remote_timeout = "30s"
}
Note: Prometheus uses service discovery to automatically detect and scrape metrics from pods. Ensure that the pod labels are properly set for accurate discovery.
Warning: If a pod is deleted or recreated, Prometheus will re-discover it only after the service discovery interval has passed. This can lead to temporary gaps in metrics collection.

3. Discovering Services

Service Discovery Method Description Notes
ServiceMonitor prometheus.io/scrape Monitors services via annotations Requires prometheus.io/port annotation
PodMonitor prometheus.io/scrape Monitors pods directly Useful for per-pod metrics
Endpoints endpointslice Discovers endpoints via EndpointSlices Recommended for large-scale services
Ingress ingress Discovers services via Ingress resources Useful for external service exposure

Example: ServiceMonitor Configuration

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: example-app-monitor
spec:
  selector:
    matchLabels:
      app: example-app
  endpoints:
  - port: http
    path: /metrics
Note: Ensure that the Prometheus instance has appropriate RBAC permissions to access Kubernetes resources.

4. Discovering Endpoints

Endpoint Description Method Notes
/api/v1/series Query for series matching a label set GET Useful for discovering metrics by labels
/api/v1/labels Query for label names GET Lists all available label names in the system
/api/v1/label/{name}/values Query for label values GET Returns all unique values for a specific label name
/api/v1/query Query for a single metric at a given time GET Allows querying for a specific metric value at a timestamp
/api/v1/query_range Query for a metric over a range of time GET Useful for generating graphs or timelines of metric data

Example: Querying for label values

curl "http://prometheus-server:9090/api/v1/label/__name__/values"

Example: Discovering series matching a label

curl "http://prometheus-server:9090/api/v1/series?match[]={__name__=~"node_.*"}"

5. Discovering Nodes

Property Description Example
prometheus.io/scrape Annotation to enable scraping of a pod or service true
prometheus.io/path Path to scrape metrics from (default: /metrics) /metrics
prometheus.io/port Port to scrape metrics from (default: 80) 9090
prometheus.io/interval Scrape interval (default: 30s) 60s

Example: Service Discovery Configuration

apiVersion: v1
kind: Service
metadata:
  name: my-app
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"
spec:
  selector:
    app: my-app
  ports:
  - port: 8080
    targetPort: 8080
Note: Ensure that the service or pod has a prometheus.io/scrape annotation set to true for Prometheus to discover and scrape it.
Warning: If prometheus.io/port is not specified, Prometheus will attempt to scrape from port 80 by default.

6. Using Ingress Discovery

Property Description Default Value Notes
ingress.name Name of the Ingress resource None Required field for discovery
ingress.namespace Namespace where the Ingress is located default Can be overridden for multi-namespace setups
ingress.labelSelector Label selector to filter Ingress resources app.kubernetes.io/name=nginx Supports standard Kubernetes label syntax

Example: Ingress Discovery Configuration

ingress:
  name: "prometheus-ingress"
  namespace: "monitoring"
  labelSelector: "app.kubernetes.io/name=prometheus"
Note: Ingress discovery requires the prometheus-operator to have appropriate RBAC permissions to list Ingress resources in the specified namespace.

Key Features of Ingress Discovery

  • Discovers Ingress resources based on label selectors
  • Supports cross-namespace discovery with proper RBAC
  • Maps Ingress paths to Prometheus scrape targets

7. Implementing Pod Annotations

Annotation Description Example Value Notes
prometheus.io/scrape Controls whether Prometheus should scrape metrics from this pod true Set to false to disable scraping
prometheus.io/path Path to scrape metrics from (default: /metrics) /metrics Used when the metrics endpoint is not at root
prometheus.io/port Port to scrape metrics from (default: 9090) 8080 Must match the port where metrics are exposed
prometheus.io/interval Scrape interval (default: 15s) 30s Overrides default scrape interval for this pod

Example: Pod with Prometheus Annotations

apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: "/metrics"
    prometheus.io/port: "8080"
spec:
  containers:
  - name: example-container
    image: example-image
    ports:
    - containerPort: 8080
Note: Annotations must be set on the Pod's metadata, not the container spec.

8. Configuring Namespace Filtering

Option Values Description
--namespace string Include only metrics from the specified namespace
--namespace.exclude string Exclude metrics from the specified namespace
--namespace.include string Include metrics from the specified namespace (alias for --namespace)

Example: Filtering Metrics by Namespace

prometheus:
  config:
    scrape_configs:
      - job_name: kubernetes-pods
        kubernetes_sd_configs:
          - role: pod
        namespace_selector:
          match_names:
            - production
            - staging
Note: The namespace_selector allows fine-grained control over which namespaces are included or excluded from scraping.
Warning: If no namespace is specified, Prometheus will scrape all namespaces by default.

9. Using Relabeling for K8s Labels

Example: Relabeling configuration for Kubernetes labels

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
    - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        target_label: __metrics_path__
        action: replace
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        target_label: __address__
        action: replace
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: app
        action: replace
      - source_labels: [__meta_kubernetes_pod_label_version]
        target_label: version
        action: replace
Relabeling Action Description Usage
keep Keeps metrics only if the regex matches Filter pods by annotation
replace Replaces target_label with value from source_labels Map labels to Prometheus metrics
drop Drops metrics if the regex matches Exclude specific labels or pods
Note: __meta_kubernetes_pod_annotation_ labels are used to configure scraping behavior in Kubernetes pods.

Key Concepts

  • source_labels: The labels to read from the Kubernetes metadata
  • target_label: The label to write to in the Prometheus metric
  • action: Determines how to process the label

10. Implementing Bearer Token Auth

Configuration Description Example
bearer_token Bearer token for authentication "my-bearer-token"
bearer_token_file Path to file containing bearer token /etc/prometheus/bearer_token

Example: Configuring Bearer Token in Prometheus

scrape_configs:
  - job_name: "protected-metrics"
    metrics_path: /metrics
    scrape_interval: 15s
    static_configs:
      - targets: ['protected-app:8080']
    bearer_token: "my-bearer-token"
Note: The bearer_token field is preferred over bearer_token_file for security reasons, as it avoids exposing tokens in files.
Warning: Ensure that bearer tokens are rotated regularly and not hardcoded in configuration files in production environments.

11. Scraping Kubelet Metrics

Configuration Option Description Default Value Notes
kubelet.metrics.path Path to scrape metrics from Kubelet /metrics Used by kubelet service to expose metrics
kubelet.metrics.port Port to scrape Kubelet metrics from 10255 Default port for Kubelet metrics endpoint
kubelet.metrics.insecure Whether to skip TLS verification when scraping false Set to true for self-signed certs or insecure environments
kubelet.metrics.basic.auth.username Username for basic auth to Kubelet "" Required if basic auth is enabled on Kubelet
kubelet.metrics.basic.auth.password Password for basic auth to Kubelet "" Required if basic auth is enabled on Kubelet

Example: Prometheus configuration to scrape Kubelet metrics

scrape_configs:
  - job_name: "kubelet"
    kubernetes_sd_configs:
      - role: node
    relabel_configs:
      - target_label: __address__
        replacement: 127.0.0.1:10255
      - target_label: __metrics_path__
        value: /metrics
Note: Ensure that the kubelet service is configured to expose metrics on port 10255. This typically requires enabling the --enable-pprof flag in Kubelet configuration.
Warning: If using insecure mode, ensure that the Kubelet endpoint is only accessible from trusted sources to avoid exposing sensitive metrics.

12. Using Service Monitors

Field Description Example
metadata.name Name of the ServiceMonitor my-app-monitor
metadata.namespace Namespace where the ServiceMonitor is defined default
spec.selector Label selector to identify target services 
{
  "matchLabels": {
    "app": "my-app"
  }
}
spec.endpoints List of endpoints to scrape 
[
  {
    "port": "http-metrics",
    "path": "/metrics"
  }
]

Example: Basic ServiceMonitor

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http-metrics
    path: /metrics
Note: The ServiceMonitor must be in the same namespace as the target services or a namespace that is allowed by the Prometheus configuration.

Was this article helpful?

Related Cheat Sheets